Slicer binary download is extremely slow - consider github for hosting packaged binaries?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Steve Pieper-2
Sounds like a good idea Andrey -- I notice there's also an API to get download statistics [1], so presumably this could be integrated with our current download stats page [2].  @mhalle any thoughts?

-Steve



On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]> wrote:
Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

lasso2

I’ve noticed that download became much slower about a few months ago. Either there have been much more downloaders lately or there has been a degradation in the infrastructure. Overall, the trend seems to be that Slicer network services are getting slower and less stable (wiki, website, downloads, …), so I would not mind outsourcing more things rather than trying to fix the current infrastructure.

 

Download from Github seems much faster and while it has a number of limits for large files (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github), “releases” feature may work. If we can capture user clicks at download.slicer.org then we could still keep collecting rich information about downloaders (github api only provides download count).

 

For me (on Queen’s University network) Slicer release download takes about 5 minutes, from github takes about 20 seconds. At home, Slicer release download takes about 15 minutes, which is starting to be considerable time.

 

Andras

 

 

From: slicer-devel [mailto:[hidden email]] On Behalf Of Steve Pieper
Sent: July 21, 2016 15:09
To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred, Ph.D. <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?

 

Sounds like a good idea Andrey -- I notice there's also an API to get download statistics [1], so presumably this could be integrated with our current download stats page [2].  @mhalle any thoughts?

 

-Steve

 

 

 

On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]> wrote:

Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

 


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Ron Kikinis
The trend seems to be going upwards.

On Jul 21, 2016, at 3:49 PM, Andras Lasso <[hidden email]> wrote:

I’ve noticed that download became much slower about a few months ago. Either there have been much more downloaders lately or there has been a degradation in the infrastructure. Overall, the trend seems to be that Slicer network services are getting slower and less stable (wiki, website, downloads, …), so I would not mind outsourcing more things rather than trying to fix the current infrastructure.
 
Download from Github seems much faster and while it has a number of limits for large files (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github), “releases” feature may work. If we can capture user clicks at download.slicer.org then we could still keep collecting rich information about downloaders (github api only provides download count).
 
For me (on Queen’s University network) Slicer release download takes about 5 minutes, from github takes about 20 seconds. At home, Slicer release download takes about 15 minutes, which is starting to be considerable time.
 
Andras
 
 
From: slicer-devel [[hidden email]] On Behalf Of Steve Pieper
Sent: July 21, 2016 15:09
To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred, Ph.D. <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?
 
Sounds like a good idea Andrey -- I notice there's also an API to get download statistics [1], so presumably this could be integrated with our current download stats page [2].  @mhalle any thoughts?
 
-Steve
 
 
 
On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]> wrote:
Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
 
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

-- 
Ron Kikinis, M.D.,
Robert Greenes Distinguished Director of Biomedical Informatics
Professor of Radiology, Harvard Medical School
Director, Surgical Planning Laboratory
http://www.spl.harvard.edu/~kikinis


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Lauren O'Donnell-3
Hi all,

I second the slowness issues for slicer.org. We spent several days improving documentation for modules over the past month, and waiting for the wiki to load and reload during iterations of help text editing became the bottleneck for completing this task. This was from BWH.

I have had estimated download times for slicer nightly of up to 40 minutes at BWH, over ethernet. When this happens I try the next day instead.

It’s nice to think Slicer’s popularity is causing this, though!

Lauren 

On Jul 21, 2016, at 3:53 PM, Kikinis, Ron,M.D. <[hidden email]> wrote:

The trend seems to be going upwards.
<Screen Shot 2016-07-21 at 3.52.31 PM.png>

On Jul 21, 2016, at 3:49 PM, Andras Lasso <[hidden email]> wrote:

I’ve noticed that download became much slower about a few months ago. Either there have been much more downloaders lately or there has been a degradation in the infrastructure. Overall, the trend seems to be that Slicer network services are getting slower and less stable (wiki, website, downloads, …), so I would not mind outsourcing more things rather than trying to fix the current infrastructure.
 
Download from Github seems much faster and while it has a number of limits for large files (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github), “releases” feature may work. If we can capture user clicks at download.slicer.org then we could still keep collecting rich information about downloaders (github api only provides download count).
 
For me (on Queen’s University network) Slicer release download takes about 5 minutes, from github takes about 20 seconds. At home, Slicer release download takes about 15 minutes, which is starting to be considerable time.
 
Andras
 
 
From: slicer-devel [[hidden email]] On Behalf Of Steve Pieper
Sent: July 21, 2016 15:09
To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred, Ph.D. <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?
 
Sounds like a good idea Andrey -- I notice there's also an API to get download statistics [1], so presumably this could be integrated with our current download stats page [2].  @mhalle any thoughts?
 
-Steve
 
 
 
On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]> wrote:
Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
 
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

-- 
Ron Kikinis, M.D.,
Robert Greenes Distinguished Director of Biomedical Informatics
Professor of Radiology, Harvard Medical School
Director, Surgical Planning Laboratory
http://www.spl.harvard.edu/~kikinis

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Isaiah Norton-2
In reply to this post by lasso2
I would be surprised if the throughput stayed un-throttled for very long -- Github does throttle, just opaquely. I don't know about issues with Releases specifically (few large projects use it), but this is in the terms of service, and it does happen at both the clone and API level (even refreshing a page too frequently still leads to the Unicorn occasionally, though rarer these days).

Releases is a free auxiliary to a free service. It is kind of hard to imagine that they want to provide max throughput for 1+ GB of inbound, and some multiple of that outbound, on a daily basis. Slicer is a "tad" larger than the typical few kB Javascript/Python library... 


On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:

I’ve noticed that download became much slower about a few months ago. Either there have been much more downloaders lately or there has been a degradation in the infrastructure. Overall, the trend seems to be that Slicer network services are getting slower and less stable (wiki, website, downloads, …), so I would not mind outsourcing more things rather than trying to fix the current infrastructure.

 

Download from Github seems much faster and while it has a number of limits for large files (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github), “releases” feature may work. If we can capture user clicks at download.slicer.org then we could still keep collecting rich information about downloaders (github api only provides download count).

 

For me (on Queen’s University network) Slicer release download takes about 5 minutes, from github takes about 20 seconds. At home, Slicer release download takes about 15 minutes, which is starting to be considerable time.

 

Andras

 

 

From: slicer-devel [mailto:[hidden email]] On Behalf Of Steve Pieper
Sent: July 21, 2016 15:09
To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred, Ph.D. <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?

 

Sounds like a good idea Andrey -- I notice there's also an API to get download statistics [1], so presumably this could be integrated with our current download stats page [2].  @mhalle any thoughts?

 

-Steve

 

 

 

On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]> wrote:

Hi,

at least at BWH, it takes on the order of 15 minutes.

I also had reports that it is that slow or even much slower from
off-BWH locations.

Should we consider using github for hosting Slicer binaries?

As an experiment, I uploaded a recent Mac binary here, and it takes
about 3 minutes to download it at BWH:
https://github.com/fedorov/Slicer/releases/tag/test-upload

The process of tagging nightly releases and uploading the binary
packages can be automated via API. The size constraint on one file is
2Gb (https://help.github.com/articles/distributing-large-binaries/).

Aside from improved performance, this would also be a nice step
towards unifying infrastructure components.

AF
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

 


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
In reply to this post by lasso2
On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
> I would be surprised if the throughput stayed un-throttled for very long --

Sounds like something we can test rather easily and transparently to
the user by swapping the download link for a few days/weeks

> Github does throttle, just opaquely. I don't know about issues with Releases
> specifically (few large projects use it), but this is in the terms of
> service, and it does happen at both the clone and API level (even refreshing
> a page too frequently still leads to the Unicorn occasionally, though rarer
> these days).
>
> Releases is a free auxiliary to a free service. It is kind of hard to
> imagine that they want to provide max throughput for 1+ GB of inbound, and
> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
> than the typical few kB Javascript/Python library...
>
>
> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>
>> I’ve noticed that download became much slower about a few months ago.
>> Either there have been much more downloaders lately or there has been a
>> degradation in the infrastructure. Overall, the trend seems to be that
>> Slicer network services are getting slower and less stable (wiki, website,
>> downloads, …), so I would not mind outsourcing more things rather than
>> trying to fix the current infrastructure.
>>
>>
>>
>> Download from Github seems much faster and while it has a number of limits
>> for large files
>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> “releases” feature may work. If we can capture user clicks at
>> download.slicer.org then we could still keep collecting rich information
>> about downloaders (github api only provides download count).
>>
>>
>>
>> For me (on Queen’s University network) Slicer release download takes about
>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> download takes about 15 minutes, which is starting to be considerable time.
>>
>>
>>
>> Andras
>>
>>
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 21, 2016 15:09
>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> Ph.D. <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> download statistics [1], so presumably this could be integrated with our
>> current download stats page [2].  @mhalle any thoughts?
>>
>>
>>
>> -Steve
>>
>>
>>
>> [1]
>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>
>>
>>
>> [2] http://download.slicer.org/download-stats/
>>
>>
>>
>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> Hi,
>>
>> at least at BWH, it takes on the order of 15 minutes.
>>
>> I also had reports that it is that slow or even much slower from
>> off-BWH locations.
>>
>> Should we consider using github for hosting Slicer binaries?
>>
>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> about 3 minutes to download it at BWH:
>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>
>> The process of tagging nightly releases and uploading the binary
>> packages can be automated via API. The size constraint on one file is
>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>
>> Aside from improved performance, this would also be a nice step
>> towards unifying infrastructure components.
>>
>> AF
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
In reply to this post by lasso2
Also, note that according to
https://help.github.com/articles/distributing-large-binaries/

"We don't limit the total size of your binary release files, nor the
bandwidth used to deliver them. "

On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
<[hidden email]> wrote:

> On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
>> I would be surprised if the throughput stayed un-throttled for very long --
>
> Sounds like something we can test rather easily and transparently to
> the user by swapping the download link for a few days/weeks
>
>> Github does throttle, just opaquely. I don't know about issues with Releases
>> specifically (few large projects use it), but this is in the terms of
>> service, and it does happen at both the clone and API level (even refreshing
>> a page too frequently still leads to the Unicorn occasionally, though rarer
>> these days).
>>
>> Releases is a free auxiliary to a free service. It is kind of hard to
>> imagine that they want to provide max throughput for 1+ GB of inbound, and
>> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
>> than the typical few kB Javascript/Python library...
>>
>>
>> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>>
>>> I’ve noticed that download became much slower about a few months ago.
>>> Either there have been much more downloaders lately or there has been a
>>> degradation in the infrastructure. Overall, the trend seems to be that
>>> Slicer network services are getting slower and less stable (wiki, website,
>>> downloads, …), so I would not mind outsourcing more things rather than
>>> trying to fix the current infrastructure.
>>>
>>>
>>>
>>> Download from Github seems much faster and while it has a number of limits
>>> for large files
>>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>>> “releases” feature may work. If we can capture user clicks at
>>> download.slicer.org then we could still keep collecting rich information
>>> about downloaders (github api only provides download count).
>>>
>>>
>>>
>>> For me (on Queen’s University network) Slicer release download takes about
>>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>>> download takes about 15 minutes, which is starting to be considerable time.
>>>
>>>
>>>
>>> Andras
>>>
>>>
>>>
>>>
>>>
>>> From: slicer-devel [mailto:[hidden email]] On Behalf
>>> Of Steve Pieper
>>> Sent: July 21, 2016 15:09
>>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>>> Ph.D. <[hidden email]>
>>> Cc: SPL Slicer Devel <[hidden email]>
>>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>>> consider github for hosting packaged binaries?
>>>
>>>
>>>
>>> Sounds like a good idea Andrey -- I notice there's also an API to get
>>> download statistics [1], so presumably this could be integrated with our
>>> current download stats page [2].  @mhalle any thoughts?
>>>
>>>
>>>
>>> -Steve
>>>
>>>
>>>
>>> [1]
>>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>>
>>>
>>>
>>> [2] http://download.slicer.org/download-stats/
>>>
>>>
>>>
>>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>>> wrote:
>>>
>>> Hi,
>>>
>>> at least at BWH, it takes on the order of 15 minutes.
>>>
>>> I also had reports that it is that slow or even much slower from
>>> off-BWH locations.
>>>
>>> Should we consider using github for hosting Slicer binaries?
>>>
>>> As an experiment, I uploaded a recent Mac binary here, and it takes
>>> about 3 minutes to download it at BWH:
>>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>>
>>> The process of tagging nightly releases and uploading the binary
>>> packages can be automated via API. The size constraint on one file is
>>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>>
>>> Aside from improved performance, this would also be a nice step
>>> towards unifying infrastructure components.
>>>
>>> AF
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with
>>> unsubscribe as the subject
>>>
>>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with
>>> unsubscribe as the subject
>>>
>>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Jean-Christophe Fillion-Robin
Hi All,


I just tried to download the Windows nightly (173MB) and it took ~4.5min over the wifi connection served by a regular DSL connection. See below.

Npte: I just asked Kitware sysadmin is anything changed regarding the bandwidth allocated with slicer.kitware.com, I will keep you posted.



In the mean time, I uploaded the windows package associated with v4.5.0-1 as a binary attachment to the corresponding release on Github.

And download took ~3.25mins


If that helps, we could definitively update download.slicer.org so that it download releases from Github instead of slicer.kitware.com. Code is available here: https://github.com/mhalle/slicer4-download



@Lauren:

   > "estimated download times for slicer nightly of up to 40 minutes at BWH, over ethernet"

   This is a long time, could you check with the IT team in charge of your network if there are any antivirus scanning downloads ?


   > waiting for the wiki to load and reload during iterations

   I also confirm this. The BWH IT has been notified and is working on the issue.


@Mike:

   I suggest you upgrade the "download.slicer.org" website to server the archive using https.


@Andras:

   > Slicer release download takes about 5 minutes,

   That sounds reasonable

   > from github takes about 20 seconds

   I suspect the university network is directly connected on the backbone network

  > 15mins from home

  For sake of comparison, could you try downloading  https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1


   
// ---------------------------------
[1] Download from slicer.kitware.com:

$ time wget http://download.slicer.org/bitstream/525502
--2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
[...]
                        100%[======================================================================================>] 172.67M   592KB/s   in 4m 23s

2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]

real    4m24.041s


\// ---------------------------------
[2] Download from  https://github.com/Slicer/Slicer/releases/download/

$ time wget https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
--2016-07-22 02:01:37--  https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
Resolving github.com (github.com)... 192.30.253.112
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
[...]
 [following]
--2016-07-22 02:01:37--  https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
[...]
Connecting to github-cloud.s3.amazonaws.com (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 183240079 (175M) [application/octet-stream]
Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’

Slicer-4.5.0-1-win-amd64.exe              100%[======================================================================================>] 174.75M   935KB/s   in 3m 14s

2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved [183240079/183240079]

real    3m14.451s



On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]> wrote:
Also, note that according to
https://help.github.com/articles/distributing-large-binaries/

"We don't limit the total size of your binary release files, nor the
bandwidth used to deliver them. "

On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
<[hidden email]> wrote:
> On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
>> I would be surprised if the throughput stayed un-throttled for very long --
>
> Sounds like something we can test rather easily and transparently to
> the user by swapping the download link for a few days/weeks
>
>> Github does throttle, just opaquely. I don't know about issues with Releases
>> specifically (few large projects use it), but this is in the terms of
>> service, and it does happen at both the clone and API level (even refreshing
>> a page too frequently still leads to the Unicorn occasionally, though rarer
>> these days).
>>
>> Releases is a free auxiliary to a free service. It is kind of hard to
>> imagine that they want to provide max throughput for 1+ GB of inbound, and
>> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
>> than the typical few kB Javascript/Python library...
>>
>>
>> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>>
>>> I’ve noticed that download became much slower about a few months ago.
>>> Either there have been much more downloaders lately or there has been a
>>> degradation in the infrastructure. Overall, the trend seems to be that
>>> Slicer network services are getting slower and less stable (wiki, website,
>>> downloads, …), so I would not mind outsourcing more things rather than
>>> trying to fix the current infrastructure.
>>>
>>>
>>>
>>> Download from Github seems much faster and while it has a number of limits
>>> for large files
>>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>>> “releases” feature may work. If we can capture user clicks at
>>> download.slicer.org then we could still keep collecting rich information
>>> about downloaders (github api only provides download count).
>>>
>>>
>>>
>>> For me (on Queen’s University network) Slicer release download takes about
>>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>>> download takes about 15 minutes, which is starting to be considerable time.
>>>
>>>
>>>
>>> Andras
>>>
>>>
>>>
>>>
>>>
>>> From: slicer-devel [mailto:[hidden email]] On Behalf
>>> Of Steve Pieper
>>> Sent: July 21, 2016 15:09
>>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>>> Ph.D. <[hidden email]>
>>> Cc: SPL Slicer Devel <[hidden email]>
>>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>>> consider github for hosting packaged binaries?
>>>
>>>
>>>
>>> Sounds like a good idea Andrey -- I notice there's also an API to get
>>> download statistics [1], so presumably this could be integrated with our
>>> current download stats page [2].  @mhalle any thoughts?
>>>
>>>
>>>
>>> -Steve
>>>
>>>
>>>
>>> [1]
>>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>>
>>>
>>>
>>> [2] http://download.slicer.org/download-stats/
>>>
>>>
>>>
>>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>>> wrote:
>>>
>>> Hi,
>>>
>>> at least at BWH, it takes on the order of 15 minutes.
>>>
>>> I also had reports that it is that slow or even much slower from
>>> off-BWH locations.
>>>
>>> Should we consider using github for hosting Slicer binaries?
>>>
>>> As an experiment, I uploaded a recent Mac binary here, and it takes
>>> about 3 minutes to download it at BWH:
>>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>>
>>> The process of tagging nightly releases and uploading the binary
>>> packages can be automated via API. The size constraint on one file is
>>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>>
>>> Aside from improved performance, this would also be a nice step
>>> towards unifying infrastructure components.
>>>
>>> AF
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with
>>> unsubscribe as the subject
>>>
>>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with
>>> unsubscribe as the subject
>>>
>>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ



--
+1 919 869 8849

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
In reply to this post by andrey.fedorov
for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL

On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:

> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>
>    > Slicer release download takes about 5 minutes,
>
>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>
> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>
> --
> +1 919 869 8849
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Steve Pieper-2
For me at home it was about 1.5 minutes to download from midas (windows nightly version, xfinity cable).

On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]> wrote:
for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL

On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>
>    > Slicer release download takes about 5 minutes,
>
>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>
> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>
> --
> <a href="tel:%2B1%20919%20869%208849" value="&#43;19198698849">+1 919 869 8849
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Dzenan Zukic
I guess that speed depends on the internet connection load of the MIDAS server.

Dženan Zukić, R&D Engineer, Kitware (Carrboro, N.C.)

On Fri, Jul 22, 2016 at 7:41 AM, Steve Pieper <[hidden email]> wrote:
For me at home it was about 1.5 minutes to download from midas (windows nightly version, xfinity cable).

On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]> wrote:
for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL

On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>
>    > Slicer release download takes about 5 minutes,
>
>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>
> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>
> --
> <a href="tel:%2B1%20919%20869%208849" value="&#43;19198698849" target="_blank">+1 919 869 8849
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Sharp, Gregory C.
In reply to this post by Steve Pieper-2

This morning at Mass Gen:

 

7:15 Midas

0:55 Github

 

Github download came from amazonaws.

Would be nice to hear from users outside North America.

 

From: slicer-devel [mailto:[hidden email]] On Behalf Of Steve Pieper
Sent: Friday, July 22, 2016 7:42 AM
To: Andrey Fedorov
Cc: SPL Slicer Devel; Gregory Rundlett
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?

 

For me at home it was about 1.5 minutes to download from midas (windows nightly version, xfinity cable).

 

On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]> wrote:

for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL


On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>

>    > Slicer release download takes about 5 minutes,
>

>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>

> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov

>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>

> --
> <a href="tel:%2B1%20919%20869%208849">+1 919 869 8849

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

 


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Isaiah Norton-2
In reply to this post by lasso2
On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov <[hidden email]> wrote:
On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
> I would be surprised if the throughput stayed un-throttled for very long --

Sounds like something we can test rather easily and transparently to
the user by swapping the download link for a few days/weeks

If you are only suggesting only release binaries, then agreed: that's easy, relatively minimal, and would be good for initial user experience. Nightlies and extensions is a different order of magnitude, given the requests-based cost structure. My back-of-envelope calculation is that hosting everything at AWS S3 prices would incur $600-1000 of cost per year (at current download rates, and depending on how much backlog was kept).
 

> Github does throttle, just opaquely. I don't know about issues with Releases
> specifically (few large projects use it), but this is in the terms of
> service, and it does happen at both the clone and API level (even refreshing
> a page too frequently still leads to the Unicorn occasionally, though rarer
> these days).
>
> Releases is a free auxiliary to a free service. It is kind of hard to
> imagine that they want to provide max throughput for 1+ GB of inbound, and
> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
> than the typical few kB Javascript/Python library...
>
>
> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>
>> I’ve noticed that download became much slower about a few months ago.
>> Either there have been much more downloaders lately or there has been a
>> degradation in the infrastructure. Overall, the trend seems to be that
>> Slicer network services are getting slower and less stable (wiki, website,
>> downloads, …), so I would not mind outsourcing more things rather than
>> trying to fix the current infrastructure.
>>
>>
>>
>> Download from Github seems much faster and while it has a number of limits
>> for large files
>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> “releases” feature may work. If we can capture user clicks at
>> download.slicer.org then we could still keep collecting rich information
>> about downloaders (github api only provides download count).
>>
>>
>>
>> For me (on Queen’s University network) Slicer release download takes about
>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> download takes about 15 minutes, which is starting to be considerable time.
>>
>>
>>
>> Andras
>>
>>
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 21, 2016 15:09
>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> Ph.D. <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> download statistics [1], so presumably this could be integrated with our
>> current download stats page [2].  @mhalle any thoughts?
>>
>>
>>
>> -Steve
>>
>>
>>
>> [1]
>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>
>>
>>
>> [2] http://download.slicer.org/download-stats/
>>
>>
>>
>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> Hi,
>>
>> at least at BWH, it takes on the order of 15 minutes.
>>
>> I also had reports that it is that slow or even much slower from
>> off-BWH locations.
>>
>> Should we consider using github for hosting Slicer binaries?
>>
>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> about 3 minutes to download it at BWH:
>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>
>> The process of tagging nightly releases and uploading the binary
>> packages can be automated via API. The size constraint on one file is
>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>
>> Aside from improved performance, this would also be a nice step
>> towards unifying infrastructure components.
>>
>> AF
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
In reply to this post by lasso2

There is no extra charge from github, they absorb the costs


On Fri, Jul 22, 2016, 09:47 Isaiah Norton <[hidden email]> wrote:
On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov <[hidden email]> wrote:
On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
> I would be surprised if the throughput stayed un-throttled for very long --

Sounds like something we can test rather easily and transparently to
the user by swapping the download link for a few days/weeks

If you are only suggesting only release binaries, then agreed: that's easy, relatively minimal, and would be good for initial user experience. Nightlies and extensions is a different order of magnitude, given the requests-based cost structure. My back-of-envelope calculation is that hosting everything at AWS S3 prices would incur $600-1000 of cost per year (at current download rates, and depending on how much backlog was kept).
 

> Github does throttle, just opaquely. I don't know about issues with Releases
> specifically (few large projects use it), but this is in the terms of
> service, and it does happen at both the clone and API level (even refreshing
> a page too frequently still leads to the Unicorn occasionally, though rarer
> these days).
>
> Releases is a free auxiliary to a free service. It is kind of hard to
> imagine that they want to provide max throughput for 1+ GB of inbound, and
> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
> than the typical few kB Javascript/Python library...
>
>
> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>
>> I’ve noticed that download became much slower about a few months ago.
>> Either there have been much more downloaders lately or there has been a
>> degradation in the infrastructure. Overall, the trend seems to be that
>> Slicer network services are getting slower and less stable (wiki, website,
>> downloads, …), so I would not mind outsourcing more things rather than
>> trying to fix the current infrastructure.
>>
>>
>>
>> Download from Github seems much faster and while it has a number of limits
>> for large files
>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> “releases” feature may work. If we can capture user clicks at
>> download.slicer.org then we could still keep collecting rich information
>> about downloaders (github api only provides download count).
>>
>>
>>
>> For me (on Queen’s University network) Slicer release download takes about
>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> download takes about 15 minutes, which is starting to be considerable time.
>>
>>
>>
>> Andras
>>
>>
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 21, 2016 15:09
>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> Ph.D. <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> download statistics [1], so presumably this could be integrated with our
>> current download stats page [2].  @mhalle any thoughts?
>>
>>
>>
>> -Steve
>>
>>
>>
>> [1]
>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>
>>
>>
>> [2] http://download.slicer.org/download-stats/
>>
>>
>>
>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> Hi,
>>
>> at least at BWH, it takes on the order of 15 minutes.
>>
>> I also had reports that it is that slow or even much slower from
>> off-BWH locations.
>>
>> Should we consider using github for hosting Slicer binaries?
>>
>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> about 3 minutes to download it at BWH:
>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>
>> The process of tagging nightly releases and uploading the binary
>> packages can be automated via API. The size constraint on one file is
>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>
>> Aside from improved performance, this would also be a nice step
>> towards unifying infrastructure components.
>>
>> AF
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Isaiah Norton-2
In reply to this post by lasso2

There is no extra charge from github, they absorb the costs

I understand that. The point is whether it is realistic (and reasonable) to expect that it stays free. For context: Github disabled binary downloads with minimal notice in 2013. That's why few large projects use Releases hosting. (The largest one I know of is their own Atom and Electron, which weigh over 300 MB -- however, they don't do nightlies).
 

On Fri, Jul 22, 2016, 09:47 Isaiah Norton <[hidden email]> wrote:
On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov <[hidden email]> wrote:
On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]> wrote:
> I would be surprised if the throughput stayed un-throttled for very long --

Sounds like something we can test rather easily and transparently to
the user by swapping the download link for a few days/weeks

If you are only suggesting only release binaries, then agreed: that's easy, relatively minimal, and would be good for initial user experience. Nightlies and extensions is a different order of magnitude, given the requests-based cost structure. My back-of-envelope calculation is that hosting everything at AWS S3 prices would incur $600-1000 of cost per year (at current download rates, and depending on how much backlog was kept).
 

> Github does throttle, just opaquely. I don't know about issues with Releases
> specifically (few large projects use it), but this is in the terms of
> service, and it does happen at both the clone and API level (even refreshing
> a page too frequently still leads to the Unicorn occasionally, though rarer
> these days).
>
> Releases is a free auxiliary to a free service. It is kind of hard to
> imagine that they want to provide max throughput for 1+ GB of inbound, and
> some multiple of that outbound, on a daily basis. Slicer is a "tad" larger
> than the typical few kB Javascript/Python library...
>
>
> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>>
>> I’ve noticed that download became much slower about a few months ago.
>> Either there have been much more downloaders lately or there has been a
>> degradation in the infrastructure. Overall, the trend seems to be that
>> Slicer network services are getting slower and less stable (wiki, website,
>> downloads, …), so I would not mind outsourcing more things rather than
>> trying to fix the current infrastructure.
>>
>>
>>
>> Download from Github seems much faster and while it has a number of limits
>> for large files
>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> “releases” feature may work. If we can capture user clicks at
>> download.slicer.org then we could still keep collecting rich information
>> about downloaders (github api only provides download count).
>>
>>
>>
>> For me (on Queen’s University network) Slicer release download takes about
>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> download takes about 15 minutes, which is starting to be considerable time.
>>
>>
>>
>> Andras
>>
>>
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 21, 2016 15:09
>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> Ph.D. <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> download statistics [1], so presumably this could be integrated with our
>> current download stats page [2].  @mhalle any thoughts?
>>
>>
>>
>> -Steve
>>
>>
>>
>> [1]
>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>
>>
>>
>> [2] http://download.slicer.org/download-stats/
>>
>>
>>
>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> Hi,
>>
>> at least at BWH, it takes on the order of 15 minutes.
>>
>> I also had reports that it is that slow or even much slower from
>> off-BWH locations.
>>
>> Should we consider using github for hosting Slicer binaries?
>>
>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> about 3 minutes to download it at BWH:
>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>
>> The process of tagging nightly releases and uploading the binary
>> packages can be automated via API. The size constraint on one file is
>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>
>> Aside from improved performance, this would also be a nice step
>> towards unifying infrastructure components.
>>
>> AF
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

lasso2
In reply to this post by Steve Pieper-2

From Queen’s university network this morning: github: 12 sec, download.slicer.org: 3 min 8 sec.

 

By the way, by moving out EMSegment and DMRI components to separate extensions we would reduce the package size by 40% (from 150MB to 90MB).

 

Andras

 

From: slicer-devel [mailto:[hidden email]] On Behalf Of Steve Pieper
Sent: July 22, 2016 7:42
To: Andrey Fedorov <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>; Gregory Rundlett <[hidden email]>
Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?

 

For me at home it was about 1.5 minutes to download from midas (windows nightly version, xfinity cable).

 

On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]> wrote:

for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL


On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>

>    > Slicer release download takes about 5 minutes,
>

>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>

> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov

>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>

> --
> <a href="tel:%2B1%20919%20869%208849">+1 919 869 8849

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

 


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

finetjul

On Fri, Jul 22, 2016 at 5:21 PM, Andras Lasso <[hidden email]> wrote:

From Queen’s university network this morning: github: 12 sec, download.slicer.org: 3 min 8 sec.

 

By the way, by moving out EMSegment and DMRI components to separate extensions we would reduce the package size by 40% (from 150MB to 90MB).

 

Andras

 

From: slicer-devel [mailto:[hidden email]] On Behalf Of Steve Pieper
Sent: July 22, 2016 7:42
To: Andrey Fedorov <[hidden email]>
Cc: SPL Slicer Devel <[hidden email]>; Gregory Rundlett <[hidden email]>


Subject: Re: [slicer-devel] Slicer binary download is extremely slow - consider github for hosting packaged binaries?

 

For me at home it was about 1.5 minutes to download from midas (windows nightly version, xfinity cable).

 

On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]> wrote:

for me the difference was 3.5 min Midas vs 1 min github, from home wifi/DSL


On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
<[hidden email]> wrote:
> Hi All,
>
>
> I just tried to download the Windows nightly (173MB) and it took ~4.5min
> over the wifi connection served by a regular DSL connection. See below.
>
> Npte: I just asked Kitware sysadmin is anything changed regarding the
> bandwidth allocated with slicer.kitware.com, I will keep you posted.
>
>
>
> In the mean time, I uploaded the windows package associated with v4.5.0-1 as
> a binary attachment to the corresponding release on Github.
>
>   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
> And download took ~3.25mins
>
>
> If that helps, we could definitively update download.slicer.org so that it
> download releases from Github instead of slicer.kitware.com. Code is
> available here: https://github.com/mhalle/slicer4-download
>
>
>
> @Lauren:
>
>    > "estimated download times for slicer nightly of up to 40 minutes at
> BWH, over ethernet"
>
>    This is a long time, could you check with the IT team in charge of your
> network if there are any antivirus scanning downloads ?
>
>
>    > waiting for the wiki to load and reload during iterations
>
>    I also confirm this. The BWH IT has been notified and is working on the
> issue.
>
>
> @Mike:
>
>    I suggest you upgrade the "download.slicer.org" website to server the
> archive using https.
>
>
> @Andras:
>

>    > Slicer release download takes about 5 minutes,
>

>    That sounds reasonable
>
>    > from github takes about 20 seconds
>
>    I suspect the university network is directly connected on the backbone
> network
>
>   > 15mins from home
>
>   For sake of comparison, could you try downloading
> https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>
>
>
> // ---------------------------------
> [1] Download from slicer.kitware.com:
>
> $ time wget http://download.slicer.org/bitstream/525502
> --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
> [...]
>
> 100%[======================================================================================>]
> 172.67M   592KB/s   in 4m 23s
>
> 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>
> real    4m24.041s
>
>
> \// ---------------------------------
> [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>
> $ time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> --2016-07-22 02:01:37--
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> Resolving github.com (github.com)... 192.30.253.112
> Connecting to github.com (github.com)|192.30.253.112|:443... connected.
> [...]
>  [following]
> --2016-07-22 02:01:37--
> https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
> [...]
> Connecting to github-cloud.s3.amazonaws.com
> (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 183240079 (175M) [application/octet-stream]
> Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>
> Slicer-4.5.0-1-win-amd64.exe
> 100%[======================================================================================>]
> 174.75M   935KB/s   in 3m 14s
>
> 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
> [183240079/183240079]
>
> real    3m14.451s
>
>
>

> On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov <[hidden email]>
> wrote:
>>
>> Also, note that according to
>> https://help.github.com/articles/distributing-large-binaries/
>>
>> "We don't limit the total size of your binary release files, nor the
>> bandwidth used to deliver them. "
>>
>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov

>> <[hidden email]> wrote:
>> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>> > wrote:
>> >> I would be surprised if the throughput stayed un-throttled for very
>> >> long --
>> >
>> > Sounds like something we can test rather easily and transparently to
>> > the user by swapping the download link for a few days/weeks
>> >
>> >> Github does throttle, just opaquely. I don't know about issues with
>> >> Releases
>> >> specifically (few large projects use it), but this is in the terms of
>> >> service, and it does happen at both the clone and API level (even
>> >> refreshing
>> >> a page too frequently still leads to the Unicorn occasionally, though
>> >> rarer
>> >> these days).
>> >>
>> >> Releases is a free auxiliary to a free service. It is kind of hard to
>> >> imagine that they want to provide max throughput for 1+ GB of inbound,
>> >> and
>> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> larger
>> >> than the typical few kB Javascript/Python library...
>> >>
>> >>
>> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]> wrote:
>> >>>
>> >>> I’ve noticed that download became much slower about a few months ago.
>> >>> Either there have been much more downloaders lately or there has been
>> >>> a
>> >>> degradation in the infrastructure. Overall, the trend seems to be that
>> >>> Slicer network services are getting slower and less stable (wiki,
>> >>> website,
>> >>> downloads, …), so I would not mind outsourcing more things rather than
>> >>> trying to fix the current infrastructure.
>> >>>
>> >>>
>> >>>
>> >>> Download from Github seems much faster and while it has a number of
>> >>> limits
>> >>> for large files
>> >>>
>> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >>> “releases” feature may work. If we can capture user clicks at
>> >>> download.slicer.org then we could still keep collecting rich
>> >>> information
>> >>> about downloaders (github api only provides download count).
>> >>>
>> >>>
>> >>>
>> >>> For me (on Queen’s University network) Slicer release download takes
>> >>> about
>> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer release
>> >>> download takes about 15 minutes, which is starting to be considerable
>> >>> time.
>> >>>
>> >>>
>> >>>
>> >>> Andras
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: slicer-devel [mailto:[hidden email]] On
>> >>> Behalf
>> >>> Of Steve Pieper
>> >>> Sent: July 21, 2016 15:09
>> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael Wilfred,
>> >>> Ph.D. <[hidden email]>
>> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> >>> consider github for hosting packaged binaries?
>> >>>
>> >>>
>> >>>
>> >>> Sounds like a good idea Andrey -- I notice there's also an API to get
>> >>> download statistics [1], so presumably this could be integrated with
>> >>> our
>> >>> current download stats page [2].  @mhalle any thoughts?
>> >>>
>> >>>
>> >>>
>> >>> -Steve
>> >>>
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >>>
>> >>>
>> >>>
>> >>> [2] http://download.slicer.org/download-stats/
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> at least at BWH, it takes on the order of 15 minutes.
>> >>>
>> >>> I also had reports that it is that slow or even much slower from
>> >>> off-BWH locations.
>> >>>
>> >>> Should we consider using github for hosting Slicer binaries?
>> >>>
>> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >>> about 3 minutes to download it at BWH:
>> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >>>
>> >>> The process of tagging nightly releases and uploading the binary
>> >>> packages can be automated via API. The size constraint on one file is
>> >>> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>> >>>
>> >>> Aside from improved performance, this would also be a nice step
>> >>> towards unifying infrastructure components.
>> >>>
>> >>> AF
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> slicer-devel mailing list
>> >>> [hidden email]
>> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >>> To unsubscribe: send email to [hidden email]
>> >>> with
>> >>> unsubscribe as the subject
>> >>>
>> >>>
>> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >>
>> >>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
>

> --
> <a href="tel:%2B1%20919%20869%208849" target="_blank">+1 919 869 8849

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ

 


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ


_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

Bill Lorensen
BillsBasement, Clifton Park, NY. Two downlaods for each.

time wget http://download.slicer.org/bitstream/525502
#1
real 10m21.602s
user 0m1.012s
sys 0m4.460s
#2
real 7m9.941s
user 0m0.964s
sys 0m4.672s

time wget https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
#1
real 1m9.926s
user 0m1.280s
sys 0m2.688s

#2
real 1m6.666s
user 0m1.100s
sys 0m2.444s


On Fri, Jul 22, 2016 at 11:29 AM, Julien Finet <[hidden email]> wrote:

> From Kitware in Europe:
>
> time wget http://download.slicer.org/bitstream/525502 -> 26 minutes
> time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> -> 28 seconds
>
> Julien.
>
> On Fri, Jul 22, 2016 at 5:21 PM, Andras Lasso <[hidden email]> wrote:
>>
>> From Queen’s university network this morning: github: 12 sec,
>> download.slicer.org: 3 min 8 sec.
>>
>>
>>
>> By the way, by moving out EMSegment and DMRI components to separate
>> extensions we would reduce the package size by 40% (from 150MB to 90MB).
>>
>>
>>
>> Andras
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 22, 2016 7:42
>> To: Andrey Fedorov <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>; Gregory Rundlett
>> <[hidden email]>
>>
>>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> For me at home it was about 1.5 minutes to download from midas (windows
>> nightly version, xfinity cable).
>>
>>
>>
>> On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> for me the difference was 3.5 min Midas vs 1 min github, from home
>> wifi/DSL
>>
>>
>> On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
>> <[hidden email]> wrote:
>> > Hi All,
>> >
>> >
>> > I just tried to download the Windows nightly (173MB) and it took ~4.5min
>> > over the wifi connection served by a regular DSL connection. See below.
>> >
>> > Npte: I just asked Kitware sysadmin is anything changed regarding the
>> > bandwidth allocated with slicer.kitware.com, I will keep you posted.
>> >
>> >
>> >
>> > In the mean time, I uploaded the windows package associated with
>> > v4.5.0-1 as
>> > a binary attachment to the corresponding release on Github.
>> >
>> >   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>> >
>> > And download took ~3.25mins
>> >
>> >
>> > If that helps, we could definitively update download.slicer.org so that
>> > it
>> > download releases from Github instead of slicer.kitware.com. Code is
>> > available here: https://github.com/mhalle/slicer4-download
>> >
>> >
>> >
>> > @Lauren:
>> >
>> >    > "estimated download times for slicer nightly of up to 40 minutes at
>> > BWH, over ethernet"
>> >
>> >    This is a long time, could you check with the IT team in charge of
>> > your
>> > network if there are any antivirus scanning downloads ?
>> >
>> >
>> >    > waiting for the wiki to load and reload during iterations
>> >
>> >    I also confirm this. The BWH IT has been notified and is working on
>> > the
>> > issue.
>> >
>> >
>> > @Mike:
>> >
>> >    I suggest you upgrade the "download.slicer.org" website to server the
>> > archive using https.
>> >
>> >
>> > @Andras:
>> >
>>
>> >    > Slicer release download takes about 5 minutes,
>> >
>>
>> >    That sounds reasonable
>> >
>> >    > from github takes about 20 seconds
>> >
>> >    I suspect the university network is directly connected on the
>> > backbone
>> > network
>> >
>> >   > 15mins from home
>> >
>> >   For sake of comparison, could you try downloading
>> > https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>> >
>> >
>> >
>> > // ---------------------------------
>> > [1] Download from slicer.kitware.com:
>> >
>> > $ time wget http://download.slicer.org/bitstream/525502
>> > --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
>> > [...]
>> >
>> >
>> > 100%[======================================================================================>]
>> > 172.67M   592KB/s   in 4m 23s
>> >
>> > 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>> >
>> > real    4m24.041s
>> >
>> >
>> > \// ---------------------------------
>> > [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>> >
>> > $ time wget
>> >
>> > https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
>> > --2016-07-22 02:01:37--
>> >
>> > https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
>> > Resolving github.com (github.com)... 192.30.253.112
>> > Connecting to github.com (github.com)|192.30.253.112|:443... connected.
>> > [...]
>> >  [following]
>> > --2016-07-22 02:01:37--
>> >
>> > https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
>> > [...]
>> > Connecting to github-cloud.s3.amazonaws.com
>> > (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
>> > HTTP request sent, awaiting response... 200 OK
>> > Length: 183240079 (175M) [application/octet-stream]
>> > Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>> >
>> > Slicer-4.5.0-1-win-amd64.exe
>> >
>> > 100%[======================================================================================>]
>> > 174.75M   935KB/s   in 3m 14s
>> >
>> > 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
>> > [183240079/183240079]
>> >
>> > real    3m14.451s
>> >
>> >
>> >
>>
>> > On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov
>> > <[hidden email]>
>> > wrote:
>> >>
>> >> Also, note that according to
>> >> https://help.github.com/articles/distributing-large-binaries/
>> >>
>> >> "We don't limit the total size of your binary release files, nor the
>> >> bandwidth used to deliver them. "
>> >>
>> >> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>>
>> >> <[hidden email]> wrote:
>> >> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton
>> >> > <[hidden email]>
>> >> > wrote:
>> >> >> I would be surprised if the throughput stayed un-throttled for very
>> >> >> long --
>> >> >
>> >> > Sounds like something we can test rather easily and transparently to
>> >> > the user by swapping the download link for a few days/weeks
>> >> >
>> >> >> Github does throttle, just opaquely. I don't know about issues with
>> >> >> Releases
>> >> >> specifically (few large projects use it), but this is in the terms
>> >> >> of
>> >> >> service, and it does happen at both the clone and API level (even
>> >> >> refreshing
>> >> >> a page too frequently still leads to the Unicorn occasionally,
>> >> >> though
>> >> >> rarer
>> >> >> these days).
>> >> >>
>> >> >> Releases is a free auxiliary to a free service. It is kind of hard
>> >> >> to
>> >> >> imagine that they want to provide max throughput for 1+ GB of
>> >> >> inbound,
>> >> >> and
>> >> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> >> larger
>> >> >> than the typical few kB Javascript/Python library...
>> >> >>
>> >> >>
>> >> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]>
>> >> >> wrote:
>> >> >>>
>> >> >>> I’ve noticed that download became much slower about a few months
>> >> >>> ago.
>> >> >>> Either there have been much more downloaders lately or there has
>> >> >>> been
>> >> >>> a
>> >> >>> degradation in the infrastructure. Overall, the trend seems to be
>> >> >>> that
>> >> >>> Slicer network services are getting slower and less stable (wiki,
>> >> >>> website,
>> >> >>> downloads, …), so I would not mind outsourcing more things rather
>> >> >>> than
>> >> >>> trying to fix the current infrastructure.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Download from Github seems much faster and while it has a number of
>> >> >>> limits
>> >> >>> for large files
>> >> >>>
>> >> >>>
>> >> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >> >>> “releases” feature may work. If we can capture user clicks at
>> >> >>> download.slicer.org then we could still keep collecting rich
>> >> >>> information
>> >> >>> about downloaders (github api only provides download count).
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> For me (on Queen’s University network) Slicer release download
>> >> >>> takes
>> >> >>> about
>> >> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer
>> >> >>> release
>> >> >>> download takes about 15 minutes, which is starting to be
>> >> >>> considerable
>> >> >>> time.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Andras
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> From: slicer-devel [mailto:[hidden email]] On
>> >> >>> Behalf
>> >> >>> Of Steve Pieper
>> >> >>> Sent: July 21, 2016 15:09
>> >> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael
>> >> >>> Wilfred,
>> >> >>> Ph.D. <[hidden email]>
>> >> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely
>> >> >>> slow -
>> >> >>> consider github for hosting packaged binaries?
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Sounds like a good idea Andrey -- I notice there's also an API to
>> >> >>> get
>> >> >>> download statistics [1], so presumably this could be integrated
>> >> >>> with
>> >> >>> our
>> >> >>> current download stats page [2].  @mhalle any thoughts?
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> -Steve
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> [1]
>> >> >>>
>> >> >>>
>> >> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> [2] http://download.slicer.org/download-stats/
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >> >>> <[hidden email]>
>> >> >>> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> at least at BWH, it takes on the order of 15 minutes.
>> >> >>>
>> >> >>> I also had reports that it is that slow or even much slower from
>> >> >>> off-BWH locations.
>> >> >>>
>> >> >>> Should we consider using github for hosting Slicer binaries?
>> >> >>>
>> >> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >> >>> about 3 minutes to download it at BWH:
>> >> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >> >>>
>> >> >>> The process of tagging nightly releases and uploading the binary
>> >> >>> packages can be automated via API. The size constraint on one file
>> >> >>> is
>> >> >>> 2Gb
>> >> >>> (https://help.github.com/articles/distributing-large-binaries/).
>> >> >>>
>> >> >>> Aside from improved performance, this would also be a nice step
>> >> >>> towards unifying infrastructure components.
>> >> >>>
>> >> >>> AF
>> >> >>> _______________________________________________
>> >> >>> slicer-devel mailing list
>> >> >>> [hidden email]
>> >> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> >>> To unsubscribe: send email to [hidden email]
>> >> >>> with
>> >> >>> unsubscribe as the subject
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> slicer-devel mailing list
>> >> >>> [hidden email]
>> >> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> >>> To unsubscribe: send email to [hidden email]
>> >> >>> with
>> >> >>> unsubscribe as the subject
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >> >>
>> >> >>
>> >> _______________________________________________
>> >> slicer-devel mailing list
>> >> [hidden email]
>> >> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> To unsubscribe: send email to [hidden email] with
>> >> unsubscribe as the subject
>> >>
>> >>
>> >> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >
>> >BillsBasement, Clifton Park, NY
time wget http://download.slicer.org/bitstream/525502
real 10m21.602s
user 0m1.012s
sys 0m4.460s

time wget https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
real 1m9.926s
user 0m1.280s
sys 0m2.688s

On Fri, Jul 22, 2016 at 11:29 AM, Julien Finet <[hidden email]> wrote:

> From Kitware in Europe:
>
> time wget http://download.slicer.org/bitstream/525502 -> 26 minutes
> time wget
> https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
> -> 28 seconds
>
> Julien.
>
> On Fri, Jul 22, 2016 at 5:21 PM, Andras Lasso <[hidden email]> wrote:
>>
>> From Queen’s university network this morning: github: 12 sec,
>> download.slicer.org: 3 min 8 sec.
>>
>>
>>
>> By the way, by moving out EMSegment and DMRI components to separate
>> extensions we would reduce the package size by 40% (from 150MB to 90MB).
>>
>>
>>
>> Andras
>>
>>
>>
>> From: slicer-devel [mailto:[hidden email]] On Behalf
>> Of Steve Pieper
>> Sent: July 22, 2016 7:42
>> To: Andrey Fedorov <[hidden email]>
>> Cc: SPL Slicer Devel <[hidden email]>; Gregory Rundlett
>> <[hidden email]>
>>
>>
>> Subject: Re: [slicer-devel] Slicer binary download is extremely slow -
>> consider github for hosting packaged binaries?
>>
>>
>>
>> For me at home it was about 1.5 minutes to download from midas (windows
>> nightly version, xfinity cable).
>>
>>
>>
>> On Fri, Jul 22, 2016 at 6:58 AM, Andrey Fedorov <[hidden email]>
>> wrote:
>>
>> for me the difference was 3.5 min Midas vs 1 min github, from home
>> wifi/DSL
>>
>>
>> On Fri, Jul 22, 2016 at 2:10 AM, Jean-Christophe Fillion-Robin
>> <[hidden email]> wrote:
>> > Hi All,
>> >
>> >
>> > I just tried to download the Windows nightly (173MB) and it took ~4.5min
>> > over the wifi connection served by a regular DSL connection. See below.
>> >
>> > Npte: I just asked Kitware sysadmin is anything changed regarding the
>> > bandwidth allocated with slicer.kitware.com, I will keep you posted.
>> >
>> >
>> >
>> > In the mean time, I uploaded the windows package associated with
>> > v4.5.0-1 as
>> > a binary attachment to the corresponding release on Github.
>> >
>> >   See https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>> >
>> > And download took ~3.25mins
>> >
>> >
>> > If that helps, we could definitively update download.slicer.org so that
>> > it
>> > download releases from Github instead of slicer.kitware.com. Code is
>> > available here: https://github.com/mhalle/slicer4-download
>> >
>> >
>> >
>> > @Lauren:
>> >
>> >    > "estimated download times for slicer nightly of up to 40 minutes at
>> > BWH, over ethernet"
>> >
>> >    This is a long time, could you check with the IT team in charge of
>> > your
>> > network if there are any antivirus scanning downloads ?
>> >
>> >
>> >    > waiting for the wiki to load and reload during iterations
>> >
>> >    I also confirm this. The BWH IT has been notified and is working on
>> > the
>> > issue.
>> >
>> >
>> > @Mike:
>> >
>> >    I suggest you upgrade the "download.slicer.org" website to server the
>> > archive using https.
>> >
>> >
>> > @Andras:
>> >
>>
>> >    > Slicer release download takes about 5 minutes,
>> >
>>
>> >    That sounds reasonable
>> >
>> >    > from github takes about 20 seconds
>> >
>> >    I suspect the university network is directly connected on the
>> > backbone
>> > network
>> >
>> >   > 15mins from home
>> >
>> >   For sake of comparison, could you try downloading
>> > https://github.com/Slicer/Slicer/releases/tag/v4.5.0-1
>> >
>> >
>> >
>> > // ---------------------------------
>> > [1] Download from slicer.kitware.com:
>> >
>> > $ time wget http://download.slicer.org/bitstream/525502
>> > --2016-07-22 01:05:35--  http://download.slicer.org/bitstream/525502
>> > [...]
>> >
>> >
>> > 100%[======================================================================================>]
>> > 172.67M   592KB/s   in 4m 23s
>> >
>> > 2016-07-22 01:09:59 (671 KB/s) - ‘525502.1’ saved [181052792/181052792]
>> >
>> > real    4m24.041s
>> >
>> >
>> > \// ---------------------------------
>> > [2] Download from  https://github.com/Slicer/Slicer/releases/download/
>> >
>> > $ time wget
>> >
>> > https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
>> > --2016-07-22 02:01:37--
>> >
>> > https://github.com/Slicer/Slicer/releases/download/v4.5.0-1/Slicer-4.5.0-1-win-amd64.exe
>> > Resolving github.com (github.com)... 192.30.253.112
>> > Connecting to github.com (github.com)|192.30.253.112|:443... connected.
>> > [...]
>> >  [following]
>> > --2016-07-22 02:01:37--
>> >
>> > https://github-cloud.s3.amazonaws.com/releases/2006302/f948f07a-4fac-11e6-9a12-f4ee372b2883.exe?X-
>> > [...]
>> > Connecting to github-cloud.s3.amazonaws.com
>> > (github-cloud.s3.amazonaws.com)|54.231.32.107|:443... connected.
>> > HTTP request sent, awaiting response... 200 OK
>> > Length: 183240079 (175M) [application/octet-stream]
>> > Saving to: ‘Slicer-4.5.0-1-win-amd64.exe’
>> >
>> > Slicer-4.5.0-1-win-amd64.exe
>> >
>> > 100%[======================================================================================>]
>> > 174.75M   935KB/s   in 3m 14s
>> >
>> > 2016-07-22 02:04:51 (923 KB/s) - ‘Slicer-4.5.0-1-win-amd64.exe’ saved
>> > [183240079/183240079]
>> >
>> > real    3m14.451s
>> >
>> >
>> >
>>
>> > On Thu, Jul 21, 2016 at 4:30 PM, Andrey Fedorov
>> > <[hidden email]>
>> > wrote:
>> >>
>> >> Also, note that according to
>> >> https://help.github.com/articles/distributing-large-binaries/
>> >>
>> >> "We don't limit the total size of your binary release files, nor the
>> >> bandwidth used to deliver them. "
>> >>
>> >> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>>
>> >> <[hidden email]> wrote:
>> >> > On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton
>> >> > <[hidden email]>
>> >> > wrote:
>> >> >> I would be surprised if the throughput stayed un-throttled for very
>> >> >> long --
>> >> >
>> >> > Sounds like something we can test rather easily and transparently to
>> >> > the user by swapping the download link for a few days/weeks
>> >> >
>> >> >> Github does throttle, just opaquely. I don't know about issues with
>> >> >> Releases
>> >> >> specifically (few large projects use it), but this is in the terms
>> >> >> of
>> >> >> service, and it does happen at both the clone and API level (even
>> >> >> refreshing
>> >> >> a page too frequently still leads to the Unicorn occasionally,
>> >> >> though
>> >> >> rarer
>> >> >> these days).
>> >> >>
>> >> >> Releases is a free auxiliary to a free service. It is kind of hard
>> >> >> to
>> >> >> imagine that they want to provide max throughput for 1+ GB of
>> >> >> inbound,
>> >> >> and
>> >> >> some multiple of that outbound, on a daily basis. Slicer is a "tad"
>> >> >> larger
>> >> >> than the typical few kB Javascript/Python library...
>> >> >>
>> >> >>
>> >> >> On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]>
>> >> >> wrote:
>> >> >>>
>> >> >>> I’ve noticed that download became much slower about a few months
>> >> >>> ago.
>> >> >>> Either there have been much more downloaders lately or there has
>> >> >>> been
>> >> >>> a
>> >> >>> degradation in the infrastructure. Overall, the trend seems to be
>> >> >>> that
>> >> >>> Slicer network services are getting slower and less stable (wiki,
>> >> >>> website,
>> >> >>> downloads, …), so I would not mind outsourcing more things rather
>> >> >>> than
>> >> >>> trying to fix the current infrastructure.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Download from Github seems much faster and while it has a number of
>> >> >>> limits
>> >> >>> for large files
>> >> >>>
>> >> >>>
>> >> >>> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>> >> >>> “releases” feature may work. If we can capture user clicks at
>> >> >>> download.slicer.org then we could still keep collecting rich
>> >> >>> information
>> >> >>> about downloaders (github api only provides download count).
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> For me (on Queen’s University network) Slicer release download
>> >> >>> takes
>> >> >>> about
>> >> >>> 5 minutes, from github takes about 20 seconds. At home, Slicer
>> >> >>> release
>> >> >>> download takes about 15 minutes, which is starting to be
>> >> >>> considerable
>> >> >>> time.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Andras
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> From: slicer-devel [mailto:[hidden email]] On
>> >> >>> Behalf
>> >> >>> Of Steve Pieper
>> >> >>> Sent: July 21, 2016 15:09
>> >> >>> To: Andrey Fedorov <[hidden email]>; Halle, Michael
>> >> >>> Wilfred,
>> >> >>> Ph.D. <[hidden email]>
>> >> >>> Cc: SPL Slicer Devel <[hidden email]>
>> >> >>> Subject: Re: [slicer-devel] Slicer binary download is extremely
>> >> >>> slow -
>> >> >>> consider github for hosting packaged binaries?
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Sounds like a good idea Andrey -- I notice there's also an API to
>> >> >>> get
>> >> >>> download statistics [1], so presumably this could be integrated
>> >> >>> with
>> >> >>> our
>> >> >>> current download stats page [2].  @mhalle any thoughts?
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> -Steve
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> [1]
>> >> >>>
>> >> >>>
>> >> >>> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> [2] http://download.slicer.org/download-stats/
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>> >> >>> <[hidden email]>
>> >> >>> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> at least at BWH, it takes on the order of 15 minutes.
>> >> >>>
>> >> >>> I also had reports that it is that slow or even much slower from
>> >> >>> off-BWH locations.
>> >> >>>
>> >> >>> Should we consider using github for hosting Slicer binaries?
>> >> >>>
>> >> >>> As an experiment, I uploaded a recent Mac binary here, and it takes
>> >> >>> about 3 minutes to download it at BWH:
>> >> >>> https://github.com/fedorov/Slicer/releases/tag/test-upload
>> >> >>>
>> >> >>> The process of tagging nightly releases and uploading the binary
>> >> >>> packages can be automated via API. The size constraint on one file
>> >> >>> is
>> >> >>> 2Gb
>> >> >>> (https://help.github.com/articles/distributing-large-binaries/).
>> >> >>>
>> >> >>> Aside from improved performance, this would also be a nice step
>> >> >>> towards unifying infrastructure components.
>> >> >>>
>> >> >>> AF
>> >> >>> _______________________________________________
>> >> >>> slicer-devel mailing list
>> >> >>> [hidden email]
>> >> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> >>> To unsubscribe: send email to [hidden email]
>> >> >>> with
>> >> >>> unsubscribe as the subject
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> slicer-devel mailing list
>> >> >>> [hidden email]
>> >> >>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> >>> To unsubscribe: send email to [hidden email]
>> >> >>> with
>> >> >>> unsubscribe as the subject
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >> >>
>> >> >>
>> >> _______________________________________________
>> >> slicer-devel mailing list
>> >> [hidden email]
>> >> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> >> To unsubscribe: send email to [hidden email] with
>> >> unsubscribe as the subject
>> >>
>> >>
>> >> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>> >
>> >
>> >
>> >
>>
>> > --
>> > +1 919 869 8849
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with
> unsubscribe as the subject
> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ



--
Unpaid intern in BillsBasement at noware dot com

>> >
>> >
>>
>> > --
>> > +1 919 869 8849
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>
>>
>>
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with
>> unsubscribe as the subject
>>
>> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>
>
>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with
> unsubscribe as the subject
> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ



--
Unpaid intern in BillsBasement at noware dot com
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
Reply | Threaded
Open this post in threaded view
|

Re: Slicer binary download is extremely slow - consider github for hosting packaged binaries?

andrey.fedorov
In reply to this post by lasso2
On Fri, Jul 22, 2016 at 10:17 AM, Isaiah Norton <[hidden email]> wrote:
>> There is no extra charge from github, they absorb the costs
>
> I understand that. The point is whether it is realistic (and reasonable) to
> expect that it stays free. For context: Github disabled binary downloads
> with minimal notice in 2013. That's why few large projects use Releases
> hosting. (The largest one I know of is their own Atom and Electron, which
> weigh over 300 MB -- however, they don't do nightlies).
>

I don't think we should make the decision based on the projections about future.

Let's look at the facts of what we know now:

 - download of Slicer binaries is annoyingly slow using current infrastructure
 - download from github appears to be order of magnitude faster
 - implementation of uploading the nightly package to github can
probably be done with an extra line to the dashboard script
 - there is zero cost in terms of maintenance, no contracts, and we
can flip the switch back any moment

The only real limitation I agree with is lack of geolocation for the
IPs, but can be addressed by tracking the download button clicks.

What is the worst that can happen if github cuts us off? No nightlies
for a day or two until we fall back to midas? We can keep archiving
binaries on midas as we've been doing it.

But I don't think they would care to cut us off. From the
Slicer-centered universe, 50K downloads per year is huge, but is it
really for Amazon or github? Compressed source code repository alone
for Slicer appears to be about 400M - this is more than a mac binary,
they are handling it alright, and I don't think Slicer is the largest
repository on github.

Unlike many other infrastructure topics (web page, wiki, bug tracker,
documentation) this one seems to be least contentious to me, since
there will be no visible change to the user interface, and the time
investment to make it happen is very minor.

An easy thing to do could be to shoot a question to github support to
describe the situation and get their feedback. I can do it.

>>
>>
>> On Fri, Jul 22, 2016, 09:47 Isaiah Norton <[hidden email]> wrote:
>>>
>>> On Thu, Jul 21, 2016 at 4:24 PM, Andrey Fedorov
>>> <[hidden email]> wrote:
>>>>
>>>> On Thu, Jul 21, 2016 at 4:22 PM, Isaiah Norton <[hidden email]>
>>>> wrote:
>>>> > I would be surprised if the throughput stayed un-throttled for very
>>>> > long --
>>>>
>>>> Sounds like something we can test rather easily and transparently to
>>>> the user by swapping the download link for a few days/weeks
>>>
>>>
>>> If you are only suggesting only release binaries, then agreed: that's
>>> easy, relatively minimal, and would be good for initial user experience.
>>> Nightlies and extensions is a different order of magnitude, given the
>>> requests-based cost structure. My back-of-envelope calculation is that
>>> hosting everything at AWS S3 prices would incur $600-1000 of cost per year
>>> (at current download rates, and depending on how much backlog was kept).
>>>
>>>
>>>> > Github does throttle, just opaquely. I don't know about issues with
>>>> > Releases
>>>> > specifically (few large projects use it), but this is in the terms of
>>>> > service, and it does happen at both the clone and API level (even
>>>> > refreshing
>>>> > a page too frequently still leads to the Unicorn occasionally, though
>>>> > rarer
>>>> > these days).
>>>> >
>>>> > Releases is a free auxiliary to a free service. It is kind of hard to
>>>> > imagine that they want to provide max throughput for 1+ GB of inbound,
>>>> > and
>>>> > some multiple of that outbound, on a daily basis. Slicer is a "tad"
>>>> > larger
>>>> > than the typical few kB Javascript/Python library...
>>>> >
>>>> >
>>>> > On Thu, Jul 21, 2016 at 3:49 PM, Andras Lasso <[hidden email]>
>>>> > wrote:
>>>> >>
>>>> >> I’ve noticed that download became much slower about a few months ago.
>>>> >> Either there have been much more downloaders lately or there has been
>>>> >> a
>>>> >> degradation in the infrastructure. Overall, the trend seems to be
>>>> >> that
>>>> >> Slicer network services are getting slower and less stable (wiki,
>>>> >> website,
>>>> >> downloads, …), so I would not mind outsourcing more things rather
>>>> >> than
>>>> >> trying to fix the current infrastructure.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Download from Github seems much faster and while it has a number of
>>>> >> limits
>>>> >> for large files
>>>> >>
>>>> >> (http://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github),
>>>> >> “releases” feature may work. If we can capture user clicks at
>>>> >> download.slicer.org then we could still keep collecting rich
>>>> >> information
>>>> >> about downloaders (github api only provides download count).
>>>> >>
>>>> >>
>>>> >>
>>>> >> For me (on Queen’s University network) Slicer release download takes
>>>> >> about
>>>> >> 5 minutes, from github takes about 20 seconds. At home, Slicer
>>>> >> release
>>>> >> download takes about 15 minutes, which is starting to be considerable
>>>> >> time.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Andras
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> From: slicer-devel [mailto:[hidden email]] On
>>>> >> Behalf
>>>> >> Of Steve Pieper
>>>> >> Sent: July 21, 2016 15:09
>>>> >> To: Andrey Fedorov <[hidden email]>; Halle, Michael
>>>> >> Wilfred,
>>>> >> Ph.D. <[hidden email]>
>>>> >> Cc: SPL Slicer Devel <[hidden email]>
>>>> >> Subject: Re: [slicer-devel] Slicer binary download is extremely slow
>>>> >> -
>>>> >> consider github for hosting packaged binaries?
>>>> >>
>>>> >>
>>>> >>
>>>> >> Sounds like a good idea Andrey -- I notice there's also an API to get
>>>> >> download statistics [1], so presumably this could be integrated with
>>>> >> our
>>>> >> current download stats page [2].  @mhalle any thoughts?
>>>> >>
>>>> >>
>>>> >>
>>>> >> -Steve
>>>> >>
>>>> >>
>>>> >>
>>>> >> [1]
>>>> >>
>>>> >> https://help.github.com/articles/getting-the-download-count-for-your-releases/
>>>> >>
>>>> >>
>>>> >>
>>>> >> [2] http://download.slicer.org/download-stats/
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Jul 21, 2016 at 2:24 PM, Andrey Fedorov
>>>> >> <[hidden email]>
>>>> >> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> at least at BWH, it takes on the order of 15 minutes.
>>>> >>
>>>> >> I also had reports that it is that slow or even much slower from
>>>> >> off-BWH locations.
>>>> >>
>>>> >> Should we consider using github for hosting Slicer binaries?
>>>> >>
>>>> >> As an experiment, I uploaded a recent Mac binary here, and it takes
>>>> >> about 3 minutes to download it at BWH:
>>>> >> https://github.com/fedorov/Slicer/releases/tag/test-upload
>>>> >>
>>>> >> The process of tagging nightly releases and uploading the binary
>>>> >> packages can be automated via API. The size constraint on one file is
>>>> >> 2Gb (https://help.github.com/articles/distributing-large-binaries/).
>>>> >>
>>>> >> Aside from improved performance, this would also be a nice step
>>>> >> towards unifying infrastructure components.
>>>> >>
>>>> >> AF
>>>> >> _______________________________________________
>>>> >> slicer-devel mailing list
>>>> >> [hidden email]
>>>> >> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> >> To unsubscribe: send email to [hidden email]
>>>> >> with
>>>> >> unsubscribe as the subject
>>>> >>
>>>> >>
>>>> >> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> slicer-devel mailing list
>>>> >> [hidden email]
>>>> >> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> >> To unsubscribe: send email to [hidden email]
>>>> >> with
>>>> >> unsubscribe as the subject
>>>> >>
>>>> >>
>>>> >> http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
>>>> >
>>>> >
>
>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/FAQ
12