[Bug 63100] New: Streaming data for browsers

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 63100] New: Streaming data for browsers

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

            Bug ID: 63100
           Summary: Streaming data for browsers
           Product: POI
           Version: 3.17-FINAL
          Hardware: PC
                OS: Mac OS X 10.1
            Status: NEW
          Severity: normal
          Priority: P2
         Component: SXSSF
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

SXSSF works as designed and manages a small memory footprint when generating
large files from a database.  But it only writes data to an output stream once
everything has been written to SXSSF. This is problematic when used in web
applications:

In our use case (our website allows users to generate Excel from the database),
generating the SXSSF on the server takes about 5 minutes. Most clients give up
within a minute (or the browser does it automatically), or the proxy times out
due to no data being sent. Some users also retry the download request. A new
request for download is initiated (while the server is busy generating the
SXSSF for a client that already gave up). This can potentially lead to DOS.

To work around this issue, I've implemented a super-streaming version of SXSSF,
a `SuperSXSSF`, that relies on `rowWriter` callback to generate row data.

With this approach our service is able to stream the generated Excel directly
to the client and, best of all, is terminated in case the user cancels the
download request.

The `SuperSXSSF` prevents both download timeouts and potential DOS, while
allowing developers all other XSSF actions (i.e. define styles) that don't take
much processing time.

Now what?



Modifications at:
https://gitlab.croptrust.org/genesys-pgr/genesys-server/tree/master/src/main/java/org/apache/poi/xssf/streaming

Use case:
https://gitlab.croptrust.org/genesys-pgr/genesys-server/blob/master/src/main/java/org/genesys2/server/service/impl/DownloadServiceImpl.java

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

--- Comment #1 from PJ Fanning <[hidden email]> ---
Your project seems useful. Could you open source your own jar (ie publish it
maven central)?

We can link to your page from our https://poi.apache.org/related-projects.html

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

--- Comment #2 from Matija Obreza <[hidden email]> ---
(In reply to PJ Fanning from comment #1)
> Your project seems useful. Could you open source your own jar (ie publish it
> maven central)?
>
> We can link to your page from our
> https://poi.apache.org/related-projects.html

The changes are implemented directly in the project
https://gitlab.croptrust.org/genesys-pgr/genesys-server (Apache v2 licensed)
because it is much simpler to trick the Java classloader to use our `.class`
files vs making a whole new jar just for the few updates I needed.

Changes to original SXSSF code are at
https://gitlab.croptrust.org/genesys-pgr/genesys-server/commits/master/src/main/java/org/apache/poi/xssf/streaming,
specifically at
https://gitlab.croptrust.org/genesys-pgr/genesys-server/commit/bac27c01a997ff8cfc4352018639e685712f3136

I've been on git for long, how do I make a merge request to your code?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

--- Comment #3 from Matija Obreza <[hidden email]> ---
Our maven artifacts are in Central
http://central.maven.org/maven2/org/genesys-pgr/

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

--- Comment #4 from PJ Fanning <[hidden email]> ---
You could fork https://github.com/apache/poi and submit a pull request there.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

--- Comment #5 from Matija Obreza <[hidden email]> ---
https://github.com/apache/poi/pull/141

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63100] Streaming data for browsers

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63100

Dominik Stadler <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]