What is batch processing?
Batch processing predefined transactions, scripts, command lines or control files in a series are automated programs or jobs running parallel on a computer. Batch processing is most suitable for tasks where a large amount of data has to be processed on a regular basis. Each program/script/transaction of work in batch processing is known as a JOB.
Limitations of on premises infrastructure during batch processing
Traditionally, Jobs used to run for certain amounts of time and scheduled for a time when computers were not busy. Often this is not the case today due to large volumes of data and corresponding required quick processing requirements.
Historically speaking, to attain the large number of compute nodes that are often required with these large data sets it required a large infrastructure footprint and often a staff of highly trained employees.
What is Azure batch?
Azure provides a platform service that schedules compute-intensive work to run on a managed collection of compute nodes and can scale up and down based on the job requirement.
In batch processing, once users submit jobs then virtual machines (compute nodes) start and run the jobs, manage any failure of nodes and then shut down the node once the jobs are finished. During this batch computing or parallel processing batch pattern is used where resources can be provisioned on request at high demand and used efficiently during job runs. This process manages multiple instance, fault domain and error handling for you.
Why Azure batch?
- Dynamically adding or removing compute nodes in a pool during job execution can save both time and money.
- Enable automatic scaling on a pool of compute nodes by associating an auto scale formula with the pool.
- Enable concurrent task execution on a pool’s compute nodes which maximizes resource usage on a smaller number of nodes within the pool.
- Helps to run large-scale parallel and high performance computing (HPC) applications.
- Less manual intervention to run a job, which makes the process error-prone.
- Lower maintenance overhead for applications
*A pool is a collection of nodes on which your application runs.
- Credit card, making a reservation or making a cash withdrawal at an ATM
- Data processing involves a continual input, process and output of data (payroll and billing systems)
- Image processing -> Resizing images, Scaling images, Converting to different image formats.
- Application for encoding your video projects into a variety of codecs and file formats
What is the difference between Azure Scheduler and Azure Web Jobs compare to Azure Batch?
Azure Scheduler is a service that allows a user to schedule recurring jobs one at a time or ongoing. It is similar to Windows Task Scheduler but this is a cloud based service. Azure scheduler trigger the job which are hosted on another provide or on premises.
Example start a backup of data every day at midnight.
Azure Web Job is used for running background process for Azure Website. An example is – such as image processing, queue processing, RSS aggregation, file maintenance, and sending emails.