Site speed issues and possible solutions

All site announcements, changes and notifications are posted here. Please check from time to time.

Moderators: parametric, Derek, Saul

User avatar
Saul
Site Admin
Site Admin
Posts: 13003
Joined: Sun Feb 01, 2004 1:00 am
Location.: UK, West Sussex
Contact:

Site speed issues and possible solutions

Unread post by Saul »

I don't need to tell any of you that the site has at times been extremely slow of late. And even moving to a VPS with dedicated resources has only helped a little.

Over the course of today, I have had various people analyze the site to see if they could figure out where the problem lays. The overriding consensus is that the site is way too bloated!

Images are too large and need optimizing for file size because they are slowing the site like the anchor of a boat dragging on the seafloor. This is my own fault. A handful of people had complained in the past about not being able to upload photos so I raised the file size limits to accommodate and opened the doors to people uploading huge files to the site. What I should have done was stuck to small file size limits and made the person who wanted to upload ensure their files were optimised.

I think there is a way to retrospectively reduce the file size of all the images uploaded to date but need to check on that. I will though be putting restrictions in place going forward.

Likewise, we cannot be a "file server". As much as I would like to host thousands of files for people to download we just don't have that sort of infrastructure and budget.

This takes a lot more thought though so I am not going to mess with what is already there. But we might have to limit file sizes going forward.

Another thing I want to try is to strip the forum back to the basics. Have only the standard phpbb theme available and remove all extensions. Not permanently I should add. This is an experiment to see if it has a positive effect on site speed.

For most of us...I think? It would be far better to have a super quick forum and maybe sacrifice a few additions than to go with all the extras and huge file sizes but with a slow loading site?

I won't do anything before the weekend and would be interested to hear your thoughts about this?

atb

Saul


Saul
Site Admin

Appreciate the forum? Buy me a coffeeImage
Subscribe To Our YouTube Channel!
Follow us on Twitter @YamahaMusicians
Follow Us On Facebook
Dowina Acoustic Guitars: Dowina Cabernet DC
Dowina Chardonnay GAC DS - Dowina Rustica GAC - Dowina Marus - Dowina Puella

Please Note: This site is at times supported by Amazon Associate and other affiliate links. Any links in my posts may lead to products on the Amazon website or other vendors websites. These will not incur any extra cost to you but may help support the running costs of the site via the generation of a small commission should you choose to purchase a product.
User avatar
EB5AGV
Member
Member
Posts: 324
Joined: Sun Feb 25, 2018 11:02 pm
Contact:

Re: Site speed issues and possible solutions

Unread post by EB5AGV »

Well, if the speed gain is noticeable, I would go for the basic forum. But I don't know exactly what that would mean for users, which extensions would be gone.

For example, I am used to host my images on my own server as lots of sites only work that way. Do you mean that kind of things?


Jose Gavila

Yamaha U3H, DX7, TX81Z, DX11, 5x SY77, 2x TG77, SY99, SPX990, SY85, A3000, 2x AN1x, EX5, 2x EX5R, EX7, FS1R, MOTIF RACK XS, MOX8, 2x Montage 6 (B & WH), Genos (and 1992' FJ1200 :mrgreen: )

Akai S5000
Arturia Keylab 61 MkII, Pigments 2, Analog Lab 4
Casio CT-X5000
Kawai K1, K1-II, VPC1
Korg M1, Wavestation, X3, Triton Rack, Triton Le 76, TR Rack
Modartt Pianoteq 6 Stage
NI Komplete 12 Ultimate Collectors Edition, Maschine Mk3, Komplete Kontrol S61 Mk2
Novation Xio 25
Roland D-50, JV-1080, U-220, XV-3080, XV-5080
Steinberg CC121, Cubase Pro 10.5, The Grand 3
StudioLogic SL88 Studio
UVI Key Suite Bundle Edition, String Machines 2
User avatar
parametric
Global Moderator
Global Moderator
Posts: 4544
Joined: Mon Jan 18, 2010 1:00 am
Location.: Near Watford, United Kingdom
Contact:

VIP

Re: Site speed issues and possible solutions

Unread post by parametric »

It's obvious we are presently carrying a huge amount of "baggage" - which I guess HAS to change if we are to remain functional

without these slowdowns.

Sure - it would be nice to become the go-to repository for all things Yamaha (for instance), but the sheer volume of resident .pdfs

.bin files, .zips of Voice Banks et Al . . . . . and that's just Yamaha . . . .

I KNOW, as I already have a large repository of such on my own machine here - which I post, when appropriate . . . . .

But it IS a considerable storage payload . . . .

As for Pictures - I have NO PROBLEM with a size limit - so long as we can see EASILY what it is . . . . .

I used to scale pics down to comply with limits, in the past . . . .

I Guess the forum can't do that automatically?? @Saul . . . . .

I think we should NOT LOSE the ability to upload pics . . . .

For certain types of topic - it's essential . . . . . .IMO

For LARGER files, I regularly use wetransfer dot com to send files to individuals . . . .

For the FREE service - It allows transfers of multiple files up to 2GB, to up to THREE recipients . . . . .

The PAID service is much more generous on transfers AND recipient numbers . . . .

I'd like to suggest the <Downloads> section remains FOR SUBSCRIBERS though . . . . as an inducement to subscribe . . . .

It presents itself as the only lever we really have ATM . . . . .?

Chris


Alesis Fusion 8SSD AND 6SSD - BOTH are 384Mb/120Gb SSD/Akai ADVANCE61/Yamaha MOXF6/1024Mb Flash Ram/Yamaha SY85/8.5mb vol/1024k non-vol/DX21/Roland MT32/Bachmann double overstrung Baby Grand Piano/Win10 Pro/Ubuntu MATE 15.0.4/iBook G4/Mac OS 10.4.6/ProTools 7.4/MBox2/M-Audio MicroTrack 24/96

NI Komplete11 Ultimate

Sector101 2x SYEMB06 / 4 x EXM-E3 128MB DRAM Module

BRAND NEW DSDD (720k) FLOPPY DISKS FOR SALE - http://www.yamahaforums.co.uk/forum/vie ... =22&t=9217

Watch out now! take care, BEWARE of the greedy leaders! They'll take you where you should not go - (George Harrison)

IT'S TRUE - "MONEY TALKS" - TO ME, IT MOSTLY SAYS "GOODBYE" ;-)
http://www.chrisnmiller.co.uk/Chris
Macke
Global Moderator
Global Moderator
Posts: 203
Joined: Thu May 30, 2019 8:03 pm
Location.: Sweden

VIP

Re: Site speed issues and possible solutions

Unread post by Macke »

I think the phpBB extentions might be a good starting point as you mentioned Saul. They might have a bigger impact on the site than one think.

I must say that I agree with Chris about the pictures being essential when it come to guides and repair threads. Setting a more resonable limit is a good suggestion which is easy to implement I assume.

Optimizing images on the fly might of course put a strain on the server once for each image upload but will probably also reduce the load and bandwith usage on every view moving forward.

I hope we don't need to strip down the forums all to much to get to a more stable/sustainable state.

I was thinking of try breathing some life into the User Sounds area as it still haven't really taken off sadly. I will postpone that until the issues at hand have been delt with.

Finally, let us know if you need assistance, Saul.


User avatar
Rara
Global Moderator
Global Moderator
Posts: 635
Joined: Sat Jul 07, 2012 4:34 pm
Location.: Australia

VIP

Re: Site speed issues and possible solutions

Unread post by Rara »

For future image uploads no more than 96dpi and Photoshop equivalent compression level of no more than 7
(roughly 65%) for JPG's and perhaps dimension limits of 1000 pixels wide.
(which I think is what I did for most of my image uploads, there may have been a few I forgot to trim)
This should help lighten the load and be big enough to illustrate guides and problems people have posted.

Rara 0-)


User avatar
dmitrins
Member
Member
Posts: 56
Joined: Fri Jul 06, 2018 7:06 am

Re: Site speed issues and possible solutions

Unread post by dmitrins »

I can clearly see in chrome devtools that there is a serious delay before involving any files into page loading process.
Here is an example (note that only 48 users are online):
Image
index.php generation time is about 16 seconds, during that time phpbb should execute several SQL queries, then arrange the results and execute other queries based on the previous results, and so on. So I think the problem may not be related to the files at all, but rather something wrong with the database (table bloating or some extension tries to execute too much queries and not optimized for doing that).
There is also some very slow ads loading process at the end:
Image


User avatar
Saul
Site Admin
Site Admin
Posts: 13003
Joined: Sun Feb 01, 2004 1:00 am
Location.: UK, West Sussex
Contact:

Re: Site speed issues and possible solutions

Unread post by Saul »

Thanks all.

To be clear, I am not talking about ditching image uploads. We will always have them. They are an integral part of the forum. It is simply to do with image "file size" as Rara pointed out. It is about being more efficient. For example if you take a high-quality photo with a DSLR or smartphone it can be 10-25 MB or even higher if it is in RAW format. That is HUGE in terms of a single upload to a forum. However, you CAN edit that photo for distribution and often get it way below 1mb without "visibly" reducing quality.

Ideally, all images need to be:

Below 500kb in file size.
Horizontal – max-width 1500px, jpeg, saved for web, and optimized.
Vertical – max-width 1000px, jpeg, saved for web, and optimized.

It is also VERY important that all images are named properly. This helps massively with our search engine ratings. So for example, if it is a picture of a Yamaha Montage 7 then name the file yamaha-montage-7. Don't leave it as 001.jpg or similar as that means nothing to search engine software. You CAN use capitals in the name but for the sake of ease, it is better to keep it all lower case. Use only letters, numbers, underscores and hyphens, nothing else. No special symbols as this will cause all sorts of problems.

When editing jpeg you want to aim for 75% image quality. Below that, you may start to notice visible degradation. The difference between 75% and 100% file size can be massive!

For some, it will be a lot of hassle because they don't know how to optimize images. So I think it would be a VERY good idea to have a sticky thread on how to optimize images for the web?

If we can nail the image file size thing it will really help a lot.

As for files, again I am not advocating we stop hosting them. Rather the larger files should be hosted outside the forum so we reduce our bandwidth usage but still provide the links. If members also keep files locally we won't risk losing any. I keep a lot of Yamaha's VST's and sample libraries locally. These are often 500 MB or more in size and apart from the copyright issues are too large to host on our server.

Regarding the extensions. I would remove all and then add back selectively and measure the impact on performance. The ones that don't slow the site down stay and the others will need to go and if possible an alternative found.

I am not sure what impact changing the forum theme would have. I "think" the PhpBB's native prosilver theme is pretty light. But I will do some more digging on that as I don't believe it is the main thing slowing the site down.

I don't think there is anything in that lot which is too drastic. We just need to be a little more attentive to what impact the things we do have on the load speed of the site. There does need to be a balance though. For me, it is not worth stripping things out just so the site loads 1 or 2 seconds faster. It needs to have a real impact to be worth the loss of something we all find useful.

As for the server itself, there is a definite bottleneck in terms of CPU resources and I am looking at ways around that now, including a move to something faster. Not something I am looking forward to but I will do whatever needs to be done to improve the speed of the site.


Saul
Site Admin

Appreciate the forum? Buy me a coffeeImage
Subscribe To Our YouTube Channel!
Follow us on Twitter @YamahaMusicians
Follow Us On Facebook
Dowina Acoustic Guitars: Dowina Cabernet DC
Dowina Chardonnay GAC DS - Dowina Rustica GAC - Dowina Marus - Dowina Puella

Please Note: This site is at times supported by Amazon Associate and other affiliate links. Any links in my posts may lead to products on the Amazon website or other vendors websites. These will not incur any extra cost to you but may help support the running costs of the site via the generation of a small commission should you choose to purchase a product.
User avatar
Rara
Global Moderator
Global Moderator
Posts: 635
Joined: Sat Jul 07, 2012 4:34 pm
Location.: Australia

VIP

Re: Site speed issues and possible solutions

Unread post by Rara »

Saul wrote: Thu Jul 30, 2020 9:24 am

When editing jpeg you want to aim for 75% image quality. Below that, you may start to notice visible degradation. The difference between 75% and 100% file size can be massive!

For some, it will be a lot of hassle because they don't know how to optimize images. So I think it would be a VERY good idea to have a sticky thread on how to optimize images for the web?

If we can nail the image file size thing it will really help a lot.
I can write up a simple guide on image optimizing/resizing,

programs like Irfanview are freeware for example (works well on linux/wine)

I have often used this to do a quick resize before I upload.

Rara 0-)


User avatar
Saul
Site Admin
Site Admin
Posts: 13003
Joined: Sun Feb 01, 2004 1:00 am
Location.: UK, West Sussex
Contact:

Re: Site speed issues and possible solutions

Unread post by Saul »

Rara wrote: Thu Jul 30, 2020 11:17 am
I can write up a simple guide on image optimizing/resizing,

programs like Irfanview are freeware for example (works well on linux/wine)

I have often used this to do a quick resize before I upload.

Rara 0-)
It would be fantastic if you could do that! Once people know how to do these things and get into the habit of reducing file sizes it all becomes so much easier :)


Saul
Site Admin

Appreciate the forum? Buy me a coffeeImage
Subscribe To Our YouTube Channel!
Follow us on Twitter @YamahaMusicians
Follow Us On Facebook
Dowina Acoustic Guitars: Dowina Cabernet DC
Dowina Chardonnay GAC DS - Dowina Rustica GAC - Dowina Marus - Dowina Puella

Please Note: This site is at times supported by Amazon Associate and other affiliate links. Any links in my posts may lead to products on the Amazon website or other vendors websites. These will not incur any extra cost to you but may help support the running costs of the site via the generation of a small commission should you choose to purchase a product.
User avatar
Saul
Site Admin
Site Admin
Posts: 13003
Joined: Sun Feb 01, 2004 1:00 am
Location.: UK, West Sussex
Contact:

Re: Site speed issues and possible solutions

Unread post by Saul »

dmitrins wrote: Thu Jul 30, 2020 8:47 am I can clearly see in chrome devtools that there is a serious delay before involving any files into page loading process.
Here is an example (note that only 48 users are online):
Image
index.php generation time is about 16 seconds, during that time phpbb should execute several SQL queries, then arrange the results and execute other queries based on the previous results, and so on. So I think the problem may not be related to the files at all, but rather something wrong with the database (table bloating or some extension tries to execute too much queries and not optimized for doing that).
There is also some very slow ads loading process at the end:
Image
There is definitely more going on that file sizes but they are not things users can do anything about. Reducing file size is something they can be proactive with and will make a difference. I also need to reduce the size of the images on the main WordPress part of the site because that also plays a part.

I agree there is a problem in the database somewhere. Probably due to failed updates in the past. And I am pretty certain at least one of the extensions is causing a problem.

I know about the advert issue. Ideally, they should all load asynchronously but some are not able to do that. This means the site waits for the advert to load before loading everything else. Which is fine if there is no bottleneck along the way! I will remove those and find another way to load them...if possible.


Saul
Site Admin

Appreciate the forum? Buy me a coffeeImage
Subscribe To Our YouTube Channel!
Follow us on Twitter @YamahaMusicians
Follow Us On Facebook
Dowina Acoustic Guitars: Dowina Cabernet DC
Dowina Chardonnay GAC DS - Dowina Rustica GAC - Dowina Marus - Dowina Puella

Please Note: This site is at times supported by Amazon Associate and other affiliate links. Any links in my posts may lead to products on the Amazon website or other vendors websites. These will not incur any extra cost to you but may help support the running costs of the site via the generation of a small commission should you choose to purchase a product.
User avatar
puremusic
Member
Member
Posts: 288
Joined: Thu Jan 31, 2019 8:48 am
Location.: USA
Contact:

Re: Site speed issues and possible solutions

Unread post by puremusic »

I'm always optimizing my images, though I don't share them here as they're not particularly music related, mostly it's just high resolution sky photography. I usually link to it off an image storage site, rather than uploading it anywhere else too. Yes they start off at 16mb or so, full sized JPGs, or more as RAWs, then I cut the resolution in half to 4k, and optimize for web at 90% jpg, usually wind up around 1mb. If I wasn't a person who really appreciated the finest detail in photography and wanted to share it, I could cut that down in half or more easily. For ordinary iamges, and stuff that's just for reference like schematics that's easy to do.

This forum reminds me of the Internet Archive, there is so much on it. That said I'm not sure one person can affordably run something like a mini Internet Archive for music on old forum software like this. If you have to restart from scratch with different forum software, and setup something streamlined, I'm sure folks'll spend a lot of time archiving and saving some of the treasure and reposting it again. It may seem like a loss but it'd generate a lot of discussion and digging up of stuff worth revisiting, combing through what to save and share again.


Roland FP-90 - Touchkeys - NS Wav5C Electric Cello - TEC BC - MIDI Expression
Kontakt - Arturia CS80 - Arturia Piano V - Sonivox Eighty-Eight - Spitfire Symphony Orchestra
User avatar
Saul
Site Admin
Site Admin
Posts: 13003
Joined: Sun Feb 01, 2004 1:00 am
Location.: UK, West Sussex
Contact:

Re: Site speed issues and possible solutions

Unread post by Saul »

puremusic wrote: Thu Jul 30, 2020 12:08 pm
This forum reminds me of the Internet Archive, there is so much on it. That said I'm not sure one person can affordably run something like a mini Internet Archive for music on old forum software like this. If you have to restart from scratch with different forum software, and setup something streamlined, I'm sure folks'll spend a lot of time archiving and saving some of the treasure and reposting it again. It may seem like a loss but it'd generate a lot of discussion and digging up of stuff worth revisiting, combing through what to save and share again.
I had actually discussed archiving this forum and starting something new but then the "archived" forum would also need hosting and people would still be downloading from it...although not uploading.

My preference is just to get this forum running a LOT faster than it currently is. I have already disabled many extensions today, one of which took the forum down! Not sure why disabling an "extension" should bring the forum down but there it is. Just another example of how it is creaking at the seams right now.

At the moment we only have 10 extensions active and 4 adverts, all of which are served asynchronously, or at least I hope they are.


Saul
Site Admin

Appreciate the forum? Buy me a coffeeImage
Subscribe To Our YouTube Channel!
Follow us on Twitter @YamahaMusicians
Follow Us On Facebook
Dowina Acoustic Guitars: Dowina Cabernet DC
Dowina Chardonnay GAC DS - Dowina Rustica GAC - Dowina Marus - Dowina Puella

Please Note: This site is at times supported by Amazon Associate and other affiliate links. Any links in my posts may lead to products on the Amazon website or other vendors websites. These will not incur any extra cost to you but may help support the running costs of the site via the generation of a small commission should you choose to purchase a product.
User avatar
dmitrins
Member
Member
Posts: 56
Joined: Fri Jul 06, 2018 7:06 am

Re: Site speed issues and possible solutions

Unread post by dmitrins »

Saul wrote: Thu Jul 30, 2020 12:07 pm There is definitely more going on that file sizes but they are not things users can do anything about. Reducing file size is something they can be proactive with and will make a difference. I also need to reduce the size of the images on the main WordPress part of the site because that also plays a part.
But the question is, how much this will improve the forum's response time? (no doubt it will save a lot of space)
I think it would be a good idea at this point to enable debug mode and see if there are any obvious bottlenecks in SQL queries.
https://t.ly/4XXu
There might be a small problem that drastically slows down the forum and eliminating it would let users enjoy this forum without a lot of additional actions and rules.


User avatar
SysExJohn
Global Moderator
Global Moderator
Posts: 912
Joined: Tue Dec 02, 2008 1:00 am
Location.: Wellingborough, Northants, UK.
Contact:

VIP

Re: Site speed issues and possible solutions

Unread post by SysExJohn »

It's still usually the first web site I visit every day (after checking my mail), and will continue to be so despite the glitches.

Keep up the good work, Saul.

We do appreciate all your efforts.

JohnG.


UW500; 2 x MU1000 + PLGs AN, DX, VL, VH & PF; SC8850; EWI4000s + VL70m; Samson Graphite 49.
Privia PX-560; AN1x; Novation X-Station 25. E-MU 02 cardbus + 1616m x 2. Finale 26; Sonar 7 PE and CbB.
XGworks v4; SQ01; SOL2. Garritan GPO5; COMB2; JABB3; IO; Organs; Steinway; World; Harps; CFX Concert Grand.
User avatar
shelly0624
Global Moderator
Global Moderator
Posts: 471
Joined: Sun Jul 14, 2019 4:13 am
Location.: United Sates

VIP

Re: Site speed issues and possible solutions

Unread post by shelly0624 »

I can write up a simple guide on image optimizing/resizing,

programs like Irfanview are freeware for example (works well on linux/wine)

I have often used this to do a quick resize before I upload.
I could really use something like this.. thank you for the offer..


Post Reply