ProCKSI

When rerun on test server (request_id=9), archive file contains:

Cannot exec tar: Argument list too long

• Archives are generated on disk, not in memory (using 'tar')br
• Still, archive.cgi cannot handle larger datasets as the connection to the web browser gets lost

• Problemsbr
  1. In the current approach, Archive::Tar::Wrapper COPIES all files to separate Temp Directory
  2. Generating the archive can take too long and the connection to the browser is lost.
• Potential Solution for Prolem 1br
  Add the Files sequentially directly to the archive without copying them using 'tar'. br
  But this would need multiple 'system' calls, each starting a new shell.
• Potential Solution for Problem 2br
  Divide functionality behind the "Download" button into two parts:
  1. Generate Archive: Generate an archive either as a background process or in the queue. Places archive into the task's home directory.
  2. Download Archive: Whenever the HTML page is refreshed, it's checked if an archive is available. If so, add this link to download the archive. Give the size of the archive.

(In r428) e - References #20
Potential solution for problem 1 added

It's good idea to prepare the archive asynchronously. It might be even more user friendly if some indication of progress would be presented to the user (like percentage of completeness) on every page refresh. You may consider using AJAX for that instead of meta-refresh.

• AJAX technology is definatively a good candidate for many of the frontend processes that either have to handle loads of data or need to wait until one certain action is performed
• It might be even worth combining AJAX in the frontend with using a special queue with high priority for all post-processing tasks, e.g. preparing an archive for download

MAJOR PROBLEM:
Size of the results might be too big to download!