Categories > TinyButStrong general >

Poor perfornance for large data merge

The forum is closed. Please use Stack Overflow for submitting new questions. Use tags: tinybutstrong , opentbs
By: DeepBlue
Date: 2010-07-09
Time: 17:55

Poor perfornance for large data merge

Hi,

For a new web application that I am building, I use Codeigniter PHP framework and TBS templating engine. So far things were ok but with growing data, TBS's show() that merges variables [var.]  into templates is performing really bad.

Often the script runs out of both maximum execution time (30 seconds) and maximum allocated memory for script (16MB). Here is sample :
- topics page with 10 topics gets takes 3.5 seconds to load.
-  the same topics page with 140 topics takes 33 seconds to load.

I have been scratching my head for several days but can't figure out the problem. Profiling tests using xdebug and Kcachegrind tell that Show() method of TBS is taking 17 seconds out of those 33.

Has anybody else encountered a similar problem and found a solution ?

Any help will be greatly appreciated.

DeepBlue
By: Anonymous
Date: 2010-07-09
Time: 21:13

Re: Poor perfornance for large data merge

Never seen pages that slow myself, but hard to say without any idea of the code... but will be glad to try to give a hand.

Post your php (with the db queries) and the template too

Pls cut and paste, psuedo-code doesn't help at all.
By: TomH
Date: 2010-07-09
Time: 21:18

Re: Poor perfornance for large data merge

That was me (Anonymous above) who forgot to ask... if you have a link to an online page that shows the results and illustrates page load time please include it.
By: Skrol29
Date: 2010-07-10
Time: 23:35

Re: Poor perfornance for large data merge

By: DeepBlue
Date: 2010-07-12
Time: 16:53

Re: Poor perfornance for large data merge

Thanks Skrl29 ,

The problem was because of [var] variables in the template. So my query result had 144 rows which had to be merged in a block. The block had only 4 [var] variables to be merged using Show() method of TBS. This caused a delay of 20 seconds for the page to load.

To fix it, I added the variables to the large array which is merged using MergeBlock and now page is normal as usual.

The merge block still takes 2 seconds to complete the merge and I am wondering if it can be optimised furthur.

@TomH : thanks for offering help but it is large template for a big project under development so it will not be possible for me to share it.
By: Skrol29
Date: 2010-07-12
Time: 23:21

Re: Poor perfornance for large data merge

Hi,

>The merge block still takes 2 seconds to complete the merge and I
> am wondering if it can be optimised furthur.

Try to avoid embedded TBS fields in you block. And see if the 2 seconds are not manly coming from the SQL query.
By: DeepBlue
Date: 2010-07-14
Time: 22:43

Re: Poor perfornance for large data merge

Now the block has no embedded TBS field and the 2 seconds are not coming from the query.

For other pages too, Merge_Block takes upto 2.2 seconds to merge a large php associative array into the block.

Here is the structure of the php array :
array ( 0 => array (
                             'a' =>1,
                             'b' => array (
                                                 'name' =>'Tom',
                                                 'age'   => 24,
                                               ),
           1=> array(......),                 
         .
         .    
        )

The array gets merged propely but merge takes a long time.
By: TomH
Date: 2010-07-15
Time: 00:28

Re: Poor perfornance for large data merge

Are you using some kind of a block subquery to populate the template?
By: TomH
Date: 2010-07-15
Time: 00:49

Re: Poor perfornance for large data merge

Ooops, pushed send too soon...

I meant to add that, with the data array you show, you can reference the name and age data directly using syntax like the following... (tested)
<tr>
<td > [blk.a]</td><td> [blk.b.name;block=tr;]</td><td align=right> [blk.b.age]</td>
</tr>
As opposed to using some king of  sub-query.

By: DeepBlue
Date: 2010-07-15
Time: 01:25

Re: Poor perfornance for large data merge

So this is what I am doing

In the codeigniter view file, there is an array $opinions obtained from a REST API call. It is merged with opTable block in one of the templates that is loaded.

$TBS->MergeBlock('opTable', 'array', $opinions);

Now in template , here a sample section of opTable block

[opTable;block=begin;]
      <table id="topic_[opTable.id]"  rpn="[opTable.position.reputation;ope=mul:100]"  style="background:url(/images/opboc_back.png) no-repeat; background-position: right bottom; border:1px solid #d3d5d2; margin:0px; width:718px; padding-bottom:40px;">
[opTable;block=end;]

The real template is pretty large but this is basic structure of the code. I can't figure out how to reduce the merge time from 2 seconds to less than 0.2 seconds. Help will be greatly appreciated.
By: TomH
Date: 2010-07-15
Time: 08:25

Re: Poor perfornance for large data merge

Try testing the merge time without the "ope=mul:100" term in the block.

If you find ope=ml to be a major contributor then try to do the pre-processing of "reputation" in the db query (in MySQL I would do "SELECT reputation*100 AS reputation ...." )

I would prefer to do all possible pre-processing in the db query, but there are other alternatives such as manipulating the data array before the merge.