Running a matrix of spring batch jobs in a Grid of Pcs or Servers to seek for DNA patterns in the whole Genome

 

Abstract :

Parallel computing is a computational method that solves a large problem by breaking it into smaller tasks and executing these tasks simultaneously on multiple processors or computing units. This approach increases efficiency and reduces computation time, enabling faster processing of large datasets and the solution of complex problems that would be too slow for traditional serial computing, which uses a single processor. Common applications of parallel computing are found in high-performance computing, artificial intelligence, smartphones, and supercomputers.

Concept :

The following is for one node one pc or one server think if we have a grid of pcs or servers

We suppose we have our Spring Batch program mvn -package

https://github.com/didipostman/CRISPR-Cas9_SpringBatchApplication

my project https://github.com/didipostman/RegEditEntiresSearch-spring-batch-gridgain-KarpRabin have a problem of multithreading concurrency issue since 2012 (GridGain springBatch partition handler)

so the solution is to use a matrix[lk] of SB (Spring Batch program) l*k programs instances allowed by disk space RAM and and evrey jdki jvm memory

ready to use for 1..n partition of DNA file or DB table calculated in advance

Create a batch file Windows or Linux allowing to copy jdk folder already installed into multiple instances

like the following

[PathToJDK1]/jdk1

[PathToJDK2]/jdk2

[PathToJDK3]/jdk3

[PathToJDK4]/jdk4

[PathToJDK5]/jdk5

.

.

.

.

.

.

[PathToJDKn]/jdkn

until n number allowed by disk storage and Ram of your PC or Server Device for multiple one jdk version supported.

On every [PathToJDKi]/jdk"i"/bin folder copy spring Batch executable program file generated by mvn -package command lets call it SB1 for the first one and then copy it into multiple SB program so we have for every

[PathToJDKi]/jdk"i"/bin/SB1

[PathToJDKi]/jdk"i"/bin/SB2

[PathToJDKi]/jdk"i"/bin/SB3

[PathToJDKi]/jdk"i"/bin/SB4

[PathToJDKi]/jdk"i"/bin/SB5 .

.

.

.

[PathToJDKi]/jdk"i"/bin/SBn

here n is the number limit allowed by a single jvm jdk"i" memory limit to execute your multiple SB java program instance.

Then launch all in parallel with different parameters for all Spring Batch instances on all JDK/bin folders like the following in a batch file commmand

for windows using start for every

[PathToJDKi]/jdki/bin/java SBl -version"il" like

start "dummyTitle" [/options] D:\path\ProgramName.exe Param1 Param2 Param3

start "dummyTitle" [/options] D:\path\ProgramName.exe Param4 Param5 Param6

so we have

start [PathToJDK1]/jdk1/bin/java SB1 -"11"

start [PathToJDK1]/jdk1/bin/java SB2 -"12"

start [PathToJDK1]/jdk1/bin/java SB3 -"13"

.

.

.

start [PathToJDK1]/jdk1/bin/java SBn -"1n"

start [PathToJDK2]/jdk2/bin/java SB1 -"21"

start [PathToJDK2]/jdk2/bin/java SB1 -"22"

start [PathToJDK2]/jdk2/bin/java SB3 -"23"

start [PathToJDK2]/jdk2/bin/java SB4 -"24"

....

start [PathToJDK2]/jdk2/bin/java SBn -"2n"

...

..

.

.

.

start [PathToJDKi]/jdki/bin/java SBl -"il"

.

..

. start [PathToJDKi]/jdki/bin/java SBn -"in"

etc

and Then execute your batch command file windows

for linux it is the same but using GNU Parallel:

Using GNU Parallel:

For advanced parallel execution, especially across multiple machines or with complex job management, GNU Parallel is a powerful tool.

parallel ::: "command1" "command2" "command3"

This runs command1, command2, and command3 in parallel. GNU Parallel offers extensive options for controlling parallelism, output handling, and error management.

-- Minds, like parachutes, function best when open. ,,,

             (o o)

/ --------oOO--(_)--OOo--------------------\

| Wadï Mami didipostman

| Github : https://www.github.com/didipostman

| e-mail : wmami@steg.com.tn / didipostman77@gmail.com

| ----------------------------------------/

Comments

Post a Comment

Popular posts from this blog

Shutdown Windows Security Threat

Image
  My World Wide Security Patch for Windows 2000/XP/7/8/10/11 and all Kind of Windows Servers 32bits/64bits. Author : Wadï Mami Publication Date : 12/07/2021 Email : wmami@steg.com.tn / didipostman77@gmail.com     Introduction Since 2015 I have contacted Microsoft for a security vulnerability which is a Dos (Disk Operating System) command. For this reason I have written .bat file which exploits this command. But MSRC (Microsoft Security Response Center) consider it as a malware and I must report it like that. I was not agree but I accepted and I reported it as a malware but they did nothing. Please visit https://github.com/didipostman/MyVirus But let me explain to you the real problem.   The problem If shutdown command is executed in Windows startup you can not use your Windows PC/Laptop anymore till an expert intervention by booting with a bootable CD/DVD or a system Flash disk and then exploring the startup Windows for a suspicious file tha...
Read more

How To Learn Languages by Keyboard

Image
  A new way to learn languages author : Wadï Mami 06/07/2020 email :  didipostman77@gmail.com Abstract : I have a problem to learn chinese. I found it difficult but I asked myself : Chinese people find learning Arabic, English, French, Deutsch etc. difficult too. So I decided to find a solution for beginners to learn languages. The new way The new method is based on learning by Keyboard and text to Speech. The teacher shows a letter to his pupils and read it twice and then asks his students to find it on the keyboard and they should press the corrspondent key and hit the baffle. Then they compare the sound response of text to Speech with what the teacher taught them. Step by Step, letter by letter and then small sentences the students will learn the basics of the languages essentially how to read. A course of spelling must be taken in parallel with what they read. Conclusion The prototype is simple to understand, download it and hit index.html and just choose a language, hit t...
Read more

Spring Batch for nanorobots

  Spring Batch is a powerful framework for batch processing in Java, but its direct application to controlling swarms of nanorobots presents significant challenges: Communication: Scalability: Nanorobot swarms will likely involve massive numbers of devices, making traditional communication methods (like Wi-Fi) impractical. Decentralization: A centralized control system would be a single point of failure. Decentralized control and peer-to-peer communication would be crucial. Real-time Constraints: Nanorobot applications often demand extremely low latency and high bandwidth for real-time coordination and data exchange. Computation: Resource Limitations: Nanorobots will have severely limited processing power and memory. Energy Efficiency: Computation must be minimized to conserve energy within the tiny devices. Safety and Ethics: Unintended Consequences: Controlling swarms of nanorobots carries significant ethical and safety implications. Robust mechanisms for...
Read more