Artificial General Intelligence (AGI)

 What is Artificial General Intelligence (AGI)? (analyticsvidhya.com)

DP-900 Microsoft Azure Data Fundamentals

 Link to AZ-900 Azure Fundamentals that I did in 2020

https://quality-agile.blogspot.com/2020/07/az-900-azure-fundamentals-microsoft.html?_sm_au_=iVV4NMVsfssn8H5bqQ2QvKH12pCN0

DP-900 Microsoft Azure Data Fundamentals

Organizations seek capturing data, storing data and analysing data. 

1. Identify common data formats
2. Describe options for storing data in files
3. Describe options for storing data in databases
4. Describe characteristics of transactional data processing solutions
5. Describe characteristics of analytical data processing solutions

1. Identify data formats

• Structured Data
• Semi-structured Data
• Unstructured Data

Difference between AI and ML

AI (Artificial Intelligence): AI is a broad field of computer science that aims to create systems capable of performing tasks that typically require human intelligence. These tasks include things like

• Ø  Understanding and processing natural language,
• Ø  Recognizing patterns in data,
• Ø  Making decisions, and even problem-solving.

AI encompasses a wide range of techniques and technologies to achieve these goals, and it's the overarching concept that drives the development of intelligent machines.

In IT terms, AI is the overarching goal of creating intelligent systems, while ML is a subset of AI that involves data-driven learning to achieve that goal. ML is often used in IT for tasks like data analysis, predictive analytics, and improving automation processes.

The key components of AI can be broadly categorized as follows:

1. Machine Learning: Machine learning is a fundamental component of AI. It involves the development of algorithms that allow computer systems to learn and improve from data, enabling them to make predictions and decisions based on patterns and trends within the data.
2. Natural Language Processing (NLP): NLP is the field that focuses on enabling machines to understand, interpret, and generate human language. It's essential for applications like language translation, chatbots, and text analysis.
3. Computer Vision: Computer vision is the branch of AI that enables computers to interpret and understand visual information from the world, including images and videos. It's used in applications like facial recognition, object detection, and autonomous vehicles.
4. Expert Systems: Expert systems are AI programs designed to mimic the decision-making abilities of a human expert in a particular domain. They use knowledge bases and inference engines to solve complex problems.
5. Robotics: Robotics involves the integration of AI into physical machines (robots) to enable them to perform tasks and interact with the physical world. AI-driven robots are used in manufacturing, healthcare, and various other industries.
6. Knowledge Representation: Knowledge representation is about how AI systems store and organize knowledge to facilitate reasoning and problem-solving. It's crucial for expert systems and reasoning tasks.
7. Neural Networks: Neural networks are a specific machine learning technique inspired by the structure of the human brain. They are used for tasks like deep learning, image and speech recognition, and reinforcement learning.
8. Planning and Decision Making: This component focuses on AI systems' ability to plan and make decisions in complex and dynamic environments. It's essential for applications like autonomous vehicles and game playing.
9. Speech and Audio Processing: This area of AI deals with the analysis and synthesis of audio data, including speech recognition and generation of human-like voices.
10. AI Ethics and Governance: With the increasing use of AI, there's a growing emphasis on ethical considerations and governance to ensure responsible AI development and use, addressing issues like bias, privacy, and transparency.
11. AI Hardware: AI often requires specialized hardware, such as Graphics Processing Units (GPUs) and Application-Specific Integrated Circuits (ASICs), to accelerate the processing of large datasets and complex AI algorithms.
12. AI Software Development Tools: A variety of software tools and libraries are used in AI development, including programming languages like Python, and frameworks like TensorFlow and PyTorch.
13. Data Management and Preprocessing: High-quality data is crucial for AI. This component involves data collection, cleaning, and preprocessing to ensure that AI systems have access to the right data.
14. AI Applications: AI is used in a wide range of applications, including virtual assistants, recommendation systems, autonomous vehicles, fraud detection, healthcare diagnosis, and much more.

These components often overlap and work together to create AI systems that can perform a wide array of tasks, ranging from simple to highly complex. AI research and development continue to evolve, leading to new components and advancements in the field.

ML (Machine Learning): Machine Learning is a specific approach within AI. It's a technique that focuses on training machines to learn from data and make predictions or decisions based on that data. Instead of writing explicit instructions for a computer program, with ML, you provide a computer system with a lot of data and algorithms that allow it to learn patterns and make predictions or decisions without being explicitly programmed for each specific task. ML is like teaching a computer to recognize spam emails by exposing it to a large dataset of emails, some of which are labeled as spam and some as not.

Dataset vs. Database

 In the context of big data, the terms "dataset" and "database" refer to different concepts and have distinct meanings. 

A dataset refers to a collection of data, while a database is a software system used to store and manage structured data. Datasets can be stored in databases, but databases can contain multiple datasets along with the necessary infrastructure to manage and manipulate the data.

Dataset: A dataset is a collection of related and structured data that is organized for a specific purpose. It represents a single unit of information that can be analyzed and processed. A dataset can consist of various types of data, such as text, numbers, images, or any other form of digital information. In the context of big data, datasets often refer to large and complex collections of data that are generated from various sources.

Datasets in big data are typically used for analysis, machine learning, and other data-driven tasks. They may include structured data (e.g., from relational databases), semi-structured data (e.g., JSON or XML documents), or unstructured data (e.g., text documents, images, videos). Datasets can be stored and accessed in various formats, such as CSV, JSON, Parquet, or databases.

Database: A database, on the other hand, is a software system used to store, manage, and organize structured data. It is a structured collection of data that is organized, indexed, and stored in a manner that allows for efficient retrieval, modification, and querying. Databases provide mechanisms for storing and retrieving data, enforcing data integrity, and supporting data manipulation operations.

Databases in the context of big data can refer to traditional relational databases, such as MySQL, Oracle, or SQL Server, as well as newer types of databases designed for big data processing, like Apache Hadoop, Apache Cassandra, or MongoDB. These big data databases are specifically designed to handle the challenges of storing and processing large volumes of data across distributed systems.

Example:

In this example, the dataset represents a collection of sales data. Each row corresponds to a separate purchase, and the columns represent different attributes of the purchase, such as the customer's name, the item purchased, the price, and the date. The dataset can be further expanded with more records to include a larger set of sales data.

From the above, it looks like a Database Table and Dataset are the same, however they are not. A dataset and a database table are similar in the sense that they both represent structured collections of data. However, there are some differences between the two:

A dataset and a database table are similar in the sense that they both represent structured collections of data. While a database table is a specific construct within a database management system, a dataset is a more general term that can encompass different types of structured data, including tables. Datasets can be more versatile, portable, and independent, while database tables are tightly coupled with the database management system and its specific rules and constraints.

1. Structure: A database table is a specific construct within a database management system (DBMS) that organizes data in rows and columns. Each column represents a specific attribute or field, while each row represents a record or entry in the table. On the other hand, a dataset is a more general term that refers to a collection of related data, which can be organized in various formats and structures, including tables. A dataset can contain multiple tables or other data structures, depending on the context.

2. Scope and Purpose: A database table is primarily used within a database management system to store and manage structured data. It is typically part of a larger database schema that includes multiple tables and relationships between them. The purpose of a database table is to provide a structured storage mechanism for data and enable efficient querying and manipulation operations. A dataset, on the other hand, can have a broader scope and purpose. It can represent a single table or a collection of tables, as well as other types of data such as files, documents, or images. Datasets are often used for analysis, machine learning, or other data-driven tasks, and they may include data from multiple sources or formats.

3. Independence: A database table is tightly linked to a specific database instance and is managed within the database management system. It is subject to the rules and constraints defined by the DBMS, such as data types, integrity constraints, and indexing. In contrast, a dataset can be more independent and portable. It can be stored and accessed in different formats and locations, such as CSV, JSON, Parquet files, or even distributed file systems. Datasets can be shared, transferred, and processed across different systems and tools without being tied to a particular database management system.

OLA and SLA

 OLA stands for Operational Level Agreement, while SLA stands for Service Level Agreement. Here's a clear and simple example to differentiate between the two:

The SLA outlines the service quality and performance targets from the customer's perspective, while the OLA defines the internal processes and responsibilities within the service provider organization to meet those targets.

Let's consider a scenario where you are a customer using a ride-hailing service like Uber or Lyft.

SLA (Service Level Agreement): The SLA is an agreement between the customer and the service provider that outlines the overall service quality and performance expectations. It defines the measurable targets and metrics that the service provider should meet. For example, the SLA may specify that the average response time for a ride request should be less than 5 minutes, or that the driver cancellation rate should be below 10%. If the service provider consistently fails to meet these targets, they would be in violation of the SLA, and there may be penalties or compensations defined in the agreement.

OLA (Operational Level Agreement): The OLA, on the other hand, focuses on the internal processes and coordination between different teams or departments within the service provider organization. It defines the responsibilities and expectations among the teams involved in delivering the service. In the context of the ride-hailing service, an OLA could specify the response time targets for the customer support team, the maintenance schedule for the vehicles, or the coordination between the dispatch team and the drivers. OLAs are not directly visible to the customers but play a crucial role in ensuring smooth operations and service delivery.