Script execution fails after increasing the number of data to be processed.

Question

When we increased the number of data to be processed in a script that had previously completed successfully, the following error occurred, and the script failed to execute.

What is the cause and what can I do about it?

OutOfMemoryError: Java heap space
OutOfMemoryError: GC overhead limit exceeded

Answer

-Cause of the error
The above error occurs when the amount of data read by HULFT Square exceeds the set heap memory.

-Countermeasures

1. Use of mass data processing and Parallel Stream Processing (PSP)
   mass data processing saves the read data in a temporary file to reduce the amount of data held in memory.
   
   Handling mass data
   https://www.hulft.com/help/en-us/HULFTSquare/Content/Data_Integration/Detailed/capacity.htm
   
   Parallel Stream Processing (PSP) divides the data to be processed into blocks, and read processing, conversion, and write processing are each processed in separate threads to reduce the amount of data held in memory.
   
   Parallel Stream Processing
   https://www.hulft.com/help/en-us/HULFTSquare/Content/Data_Integration/Detailed/parallel_streaming_processing.htm
   
   
2. Increasing the memory of Integrate Service
   Delete the currently used Integrate Service and create a new one with additional memory.
   Note that if you delete the Integrate Service, the service references of the associated jobs will be cleared, so you will need to re-specify the service after re-creation.
   Please change the settings from the Edit Job screen.

If the above measures do not resolve the error, please contact our Technical Support Center.

Related FAQs

Heap size in HULFT Square
https://support.square.hulft.com/hc/articles/7716929801492

How is the operation of the service affected when an OutOfMemoryError occurs?
https://support.square.hulft.com/hc/articles/9465225443732