Mahz's Dev Journal

[Mahz]

Latest push: - Implement ability for staff to move topics with an optional redirect.[1] - Removed all database queries from the homepage so that it loads almost instantly (always a cache hit).[2] Technical notes for the badass hackers: [1]: The "redirect" (on by default) is a "Topic moved" papertrail message that shows up in place of the topic on the origin forum so that moved topics don't just vanish. What seems like an easy feature ended up taking a while because I basically had to change from simply sorting topics by latest_post_id to sorting topics by moved_at (timestamp) and latest_post_at (timestamp). I struggled to do that sort performantly til my friend pointed out that I can create an index on the expression (topic.moved_at OR topic.latest_post_at). Blew my mind. [2]: I've been slowly moving database queries to a simple cache object I created that refreshes each cache item at some arbitrary interval in the background (source). That way cache reads are always instant. By caching the homepage's list of categories, forums, and latest posters (updated every 10 seconds), I reduced the homepage load time from 20-50ms to <1ms on localhost. So far I've spent the rest of my time working on the "Hide" system. I'm essentially trying to replace all destructive deletion with a system of is_hidden = true flags. For one, that will make all "deletes" reversible. There really isn't a good reason for a forum in 2015 to delete data, but I also just want to set the stage for user-level deletion. It's pretty hard though. Next weekend I will pick more enjoyable features to work on for sanity.

[Captain Jordan]

[2]: I've been slowly moving database queries to a simple cache object I created that refreshes each cache item at some arbitrary interval in the background (source). That way cache reads are always instant. By caching the homepage's list of categories, forums, and latest posters (updated every 10 seconds), I reduced the homepage load time from 20-50ms to <1ms on localhost.

@Mahz

I'm laughing so hard when reading this post. It's nothing you said (well, no, it's everything you said), it's just that... After you posted this, I started brainstorming with a friend of mine via instant messenger. My thoughts were:

Me: okay, so I'm curious about something Me: in a normal forum, when you load a page (like the main index) it tells you how many posts are in a particular forum, or other such statistics Me: behind the scenes, every time you load the page, the database is being queried for this data and grabbing its most current numbers Me: so here's my curiosity Me: what would happen if, during, say, the act of posting a new topic, the forum simultaneously updated the database and queried the new statistic for that post Me: then recorded it on a flat file with just basic stats Me: then page loads would pull from that file rather than the db Me: would the cheaper costs to the database outweigh the potential problems (deadlocking, overwrites, page loading errors, etc)? Friend: dunno Me: 'twould be interesting to experiment if it hasn't already been tried Me: I'm thinking like the basic stats Me: like stuff on the main page of the forum, which is loaded most frequently Me: so basic post counts for indidvidual boardstotal member numbers Me: stuff that doesn't need to be updated by the second

As you can see, my friend was extremely helpful with this brainstorm. It went nowhere. But now I see you effectively doing the same thing. And it makes me happy, @Mahz.

[Mahz]

@Captain Jordan: Haha, nice. You stumbled upon the concept of incremental local cache. Though the nice thing about keeping it in memory (as opposed to file I/O) is that you get atomic reads & writes for free (x = 42).

[Dinh AaronMk]

Hey-yo, would there be some way to fix thread URLs for when you change the topic title? Or do some redirect thing? One thing me and my friends like or try to do is do post catalogs as we RP. That way we have a index of posts and brief descriptions of what was going on in that, with a link back to that post. However with title changes this has broken and every time I change the thread title we need to go through and fix our cataloging. I like the ability to change titles, brings back for me the era of goofy sub-titles. But it busts the way we link things.

[LegendBegins]

I could only do something like that when I implement image uploads instead of the current hack. That way I could do server-side processing and, for example, output two different resolutions which will display depending on your device rez.

@Mahz

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

Hey-yo, would there be some way to fix thread URLs for when you change the topic title? Or do some redirect thing? One thing me and my friends like or try to do is do post catalogs as we RP. That way we have a index of posts and brief descriptions of what was going on in that, with a link back to that post. However with title changes this has broken and every time I change the thread title we need to go through and fix our cataloging. I like the ability to change titles, brings back for me the era of goofy sub-titles. But it busts the way we link things.

@Dinh AaronMk

Changing the title shouldn't break links. All that matters in the link is the topicid (for example, 75056 for this topic). The title is added in after, and has no impact on the resulting page. Compare: http://www.roleplayerguild.com/topics/75056-mahzs-dev-journal http://www.roleplayerguild.com/topics/75056 http://www.roleplayerguild.com/topics/75056-internet-tough-guy-was-here

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

@LegendBegins

I'm fairly sure the guild doesn't actually store any images, it just points to where an image file is stored (I believe this is how websites work – I don't work on web much (read: the guild is the first real web coding I've done), so I'm not certain on that.) so it can't do anything to the image.

[Dinh AaronMk]

Hey-yo, would there be some way to fix thread URLs for when you change the topic title? Or do some redirect thing? One thing me and my friends like or try to do is do post catalogs as we RP. That way we have a index of posts and brief descriptions of what was going on in that, with a link back to that post. However with title changes this has broken and every time I change the thread title we need to go through and fix our cataloging. I like the ability to change titles, brings back for me the era of goofy sub-titles. But it busts the way we link things.

@Dinh AaronMk

Changing the title shouldn't break links. All that matters in the link is the topicid (for example, 75056 for this topic). The title is added in after, and has no impact on the resulting page. Compare: http://www.roleplayerguild.com/topics/75056-mahzs-dev-journal http://www.roleplayerguild.com/topics/75056 http://www.roleplayerguild.com/topics/75056-internet-tough-guy-was-here

@TheMaster99

Oh, so it must have been fixed prior. Whenever we click on the links to get to that post it's redirected us back to the first page of IC, and not the post we wanted. So we may need to redo our catalogs again.

[LegendBegins]

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

@LegendBegins

I'm fairly sure the guild doesn't actually store any images, it just points to where an image file is stored (I believe this is how websites work – I don't work on web much (read: the guild is the first real web coding I've done), so I'm not certain on that.) so it can't do anything to the image.

@TheMaster99

I'm aware that it doesn't store them, but it is completely possible to edit them. You just run the image through a "filter" of sorts before output, and it displays it according to the commands. For example, Mahz has a hack that alters profile pics images larger than 150 X 150 to attempt to display them correctly. You're correct that it doesn't do anything to the image itself, but it can change the display.

[Captain Jordan]

Hey-yo, would there be some way to fix thread URLs for when you change the topic title? Or do some redirect thing? One thing me and my friends like or try to do is do post catalogs as we RP. That way we have a index of posts and brief descriptions of what was going on in that, with a link back to that post. However with title changes this has broken and every time I change the thread title we need to go through and fix our cataloging. I like the ability to change titles, brings back for me the era of goofy sub-titles. But it busts the way we link things.

@Dinh AaronMk

Changing the title shouldn't break links. All that matters in the link is the topicid (for example, 75056 for this topic). The title is added in after, and has no impact on the resulting page. Compare: http://www.roleplayerguild.com/topics/75056-mahzs-dev-journal http://www.roleplayerguild.com/topics/75056 http://www.roleplayerguild.com/topics/75056-internet-tough-guy-was-here

@TheMaster99

Oh, so it must have been fixed prior. Whenever we click on the links to get to that post it's redirected us back to the first page of IC, and not the post we wanted. So we may need to redo our catalogs again.

@Dinh AaronMk

Are you sure you're copying the link from the link icon (it's to the right of Raw at the upper-right corner of every post) and not the topic link? Post links look like this: http://www.roleplayerguild.com/posts/2404697 Topic links look like this: http://www.roleplayerguild.com/topics/75056-Something-Something-Title-Side Notice the word post or topic before the numbers.

[Captain Jordan]

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

@LegendBegins

I'm fairly sure the guild doesn't actually store any images, it just points to where an image file is stored (I believe this is how websites work – I don't work on web much (read: the guild is the first real web coding I've done), so I'm not certain on that.) so it can't do anything to the image.

@TheMaster99

I'm aware that it doesn't store them, but it is completely possible to edit them. You just run the image through a "filter" of sorts before output, and it displays it according to the commands. For example, Mahz has a hack that alters profile pics images larger than 150 X 150 to attempt to display them correctly. You're correct that it doesn't do anything to the image itself, but it can change the display.

@LegendBegins

PHP has a couple modules for this, GraphicsMagick and ImageMagick. I think Node.js has a library that will interface with the PHP modules to do this. So long as the two (PHP and Node.js modules) are installed, then it's probably just a matter of taking the time, reading the documentation, and doing it. Someone with the javascript skills and wherewithal could probably write a patch and submit a pull request if they really wanted it done quickly.

[LegendBegins]

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

@LegendBegins

I'm fairly sure the guild doesn't actually store any images, it just points to where an image file is stored (I believe this is how websites work – I don't work on web much (read: the guild is the first real web coding I've done), so I'm not certain on that.) so it can't do anything to the image.

@TheMaster99

I'm aware that it doesn't store them, but it is completely possible to edit them. You just run the image through a "filter" of sorts before output, and it displays it according to the commands. For example, Mahz has a hack that alters profile pics images larger than 150 X 150 to attempt to display them correctly. You're correct that it doesn't do anything to the image itself, but it can change the display.

@LegendBegins

PHP has a couple modules for this, GraphicsMagick and ImageMagick. I think Node.js has a library that will interface with the PHP modules to do this. So long as the two (PHP and Node.js modules) are installed, then it's probably just a matter of taking the time, reading the documentation, and doing it. Someone with the javascript skills and wherewithal could probably write a patch and submit a pull request if they really wanted it done quickly.

@Captain Jordan

Once I've finished my computer repairs, I may go ahead and have a crack at it myself.

The Guild already supports high res image embedding. What prevents using the same concept for profile pictures, and then have the server scale it down, and perform all the necessary alternations (such as the displayed resolution, as you mentioned)? For instance, is there anything that would make it impossible for the server to take these image URls, make the alterations server-wise, as if it were a local file, and display the altered output?

@LegendBegins

I'm fairly sure the guild doesn't actually store any images, it just points to where an image file is stored (I believe this is how websites work – I don't work on web much (read: the guild is the first real web coding I've done), so I'm not certain on that.) so it can't do anything to the image.

@TheMaster99

I'm aware that it doesn't store them, but it is completely possible to edit them. You just run the image through a "filter" of sorts before output, and it displays it according to the commands. For example, Mahz has a hack that alters profile pics images larger than 150 X 150 to attempt to display them correctly. You're correct that it doesn't do anything to the image itself, but it can change the display.

@LegendBegins

PHP has a couple modules for this, GraphicsMagick and ImageMagick. I think Node.js has a library that will interface with the PHP modules to do this. So long as the two (PHP and Node.js modules) are installed, then it's probably just a matter of taking the time, reading the documentation, and doing it. Someone with the javascript skills and wherewithal could probably write a patch and submit a pull request if they really wanted it done quickly.

@Captain Jordan

Once I've finished my computer repairs, I may go ahead and have a crack at it myself.

@LegendBegins

I'm 99% sure that PHP isn't used at all on the guild, so you'll probably have to find a purely node.js way to do it.

[Dinh AaronMk]

Hey-yo, would there be some way to fix thread URLs for when you change the topic title? Or do some redirect thing? One thing me and my friends like or try to do is do post catalogs as we RP. That way we have a index of posts and brief descriptions of what was going on in that, with a link back to that post. However with title changes this has broken and every time I change the thread title we need to go through and fix our cataloging. I like the ability to change titles, brings back for me the era of goofy sub-titles. But it busts the way we link things.

@Dinh AaronMk

Changing the title shouldn't break links. All that matters in the link is the topicid (for example, 75056 for this topic). The title is added in after, and has no impact on the resulting page. Compare: http://www.roleplayerguild.com/topics/75056-mahzs-dev-journal http://www.roleplayerguild.com/topics/75056 http://www.roleplayerguild.com/topics/75056-internet-tough-guy-was-here

@TheMaster99

Oh, so it must have been fixed prior. Whenever we click on the links to get to that post it's redirected us back to the first page of IC, and not the post we wanted. So we may need to redo our catalogs again.

@Dinh AaronMk

Are you sure you're copying the link from the link icon (it's to the right of Raw at the upper-right corner of every post) and not the topic link? Post links look like this: http://www.roleplayerguild.com/posts/2404697 Topic links look like this: http://www.roleplayerguild.com/topics/75056-Something-Something-Title-Side Notice the word post or topic before the numbers.

@Captain Jordan

This is how our catalog URLs are coming out in the URL box. http://www.roleplayerguild.com/topics/19063-precipice-of-war-there-be-spaniards-in-my-coffee/ic#post-1272608 Yet, this one works. http://www.roleplayerguild.com/topics/19063-precipice-of-war-there-be-spaniards-in-my-coffee/ic#post-508933

http://www.roleplayerguild.com/topics/19063-precipice-of-war-there-be-spaniards-in-my-coffee/ic?page=4#post-1272608 The link you were using is trying to read postid 1272608, on page 1 (since none is specified). This post doesn't exist on that page, thus it doesn't scroll to it. Somehow you managed to get a link that didn't have the ?page=4 - I'm not sure how, because the link button works perfectly fine for me.

[Captain Jordan]

I'm 99% sure that PHP isn't used at all on the guild, so you'll probably have to find a purely node.js way to do it.

@TheMaster99

Just because it isn't used doesn't mean it can't be installed on the server. There is no pure node.js way to do it, it would have to be done by a server-side programming language, not a scripting language like JS. At least if you wanted true rescaling (so it creates and stores an image of the correct size on the server) and not just resizing a larger image.

Are you sure you're copying the link from the link icon (it's to the right of Raw at the upper-right corner of every post) and not the topic link? Post links look like this: http://www.roleplayerguild.com/posts/2404697 Topic links look like this: http://www.roleplayerguild.com/topics/75056-Something-Something-Title-Side Notice the word post or topic before the numbers.

@Captain Jordan

This is how our catalog URLs are coming out in the URL box. http://www.roleplayerguild.com/topics/19063-precipice-of-war-there-be-spaniards-in-my-coffee/ic#post-1272608 Yet, this one works. http://www.roleplayerguild.com/topics/19063-precipice-of-war-there-be-spaniards-in-my-coffee/ic#post-508933

@Dinh AaronMk

If you simply right-click on the link icon, and click on Copy Link Address/Location in the menu. (If you're on Internet Explorer, the menu item is Copy Shortcut.) The link you get that way will always redirect to the proper page, even if the pages or titles change.

Just because it isn't used doesn't mean it can't be installed on the server. There is no pure node.js way to do it, it would have to be done by a server-side programming language, not a scripting language like JS. At least if you wanted true rescaling (so it creates and stores an image of the correct size on the server) and not just resizing a larger image.

@Captain Jordan

Yes, but why would you install something just to implement one feature that will be deprecated once Mahz decides its time to add picture uploading? No offense but I don't see the logic behind it.

[Captain Jordan]

Just because it isn't used doesn't mean it can't be installed on the server. There is no pure node.js way to do it, it would have to be done by a server-side programming language, not a scripting language like JS. At least if you wanted true rescaling (so it creates and stores an image of the correct size on the server) and not just resizing a larger image.

@Captain Jordan

Yes, but why would you install something just to implement one feature that will be deprecated once Mahz decides its time to add picture uploading? No offense but I don't see the logic behind it.

@TheMaster99

As I mentioned, it doesn't have to be done with PHP. It does have to be done with a server-size language like PHP. Java, .NET, PHP, Ruby, etc., any of these would work. I'm not certain how uploading the image is going to change much, HTML, CSS and Javascript don't have the capabilities to change and store modified images on the server. Something else will have to be used, unless Mahz has a trick up his sleeve that I've never heard of. ETA: No offense taken. It's just a friendly discussion. :)

Just because it isn't used doesn't mean it can't be installed on the server. There is no pure node.js way to do it, it would have to be done by a server-side programming language, not a scripting language like JS. At least if you wanted true rescaling (so it creates and stores an image of the correct size on the server) and not just resizing a larger image.

@Captain Jordan

Yes, but why would you install something just to implement one feature that will be deprecated once Mahz decides its time to add picture uploading? No offense but I don't see the logic behind it.

@TheMaster99

As I mentioned, it doesn't have to be done with PHP. It does have to be done with a server-size language like PHP. Java, .NET, PHP, Ruby, etc., any of these would work. I'm not certain how uploading the image is going to change much, HTML, CSS and Javascript don't have the capabilities to change and store modified images on the server. Something else will have to be used, unless Mahz has a trick up his sleeve that I've never heard of. ETA: No offense taken. It's just a friendly discussion. :)

@Captain Jordan

http://aheckmann.github.io/gm/ I found this with a simple google search, and there are most likely other options out there. Definitely don't need to install a server-side language for it :D

[Captain Jordan]

http://aheckmann.github.io/gm/ I found this with a simple google search, and there are most likely other options out there. Definitely don't need to install a server-side language for it :D

@TheMaster99

That's the node.js module I was referring to. The dependencies are clear, the executables for GM or IM. You're right, it doesn't take the full language, but these are built in a programming language. Client-side scripting languages just can't do the math required to resize images.