This forum is in READ-ONLY mode.
You can look around, but if you want to ask a new question, please use the new forum.
Home » support » General discussion » Paging with large results set
Paging with large results set [message #10304] Thu, 17 August 2006 12:01 Go to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
Hi All,

I'm currently using pagination successfully on an archive system, however due to the number of records being input (5000+) I've noticed that performance has degraded somewha. As such, I was wondering if it would be possible to improve the performance of the pager?

The backend database is MSSQL 2000 and I've limited the pager to show only 8 recorrds, but it seems to take an awfully long time to load.

Any suggestions would be greatly appreciated.

Re: Paging with large results set [message #10347 is a reply to message #10304 ] Thu, 17 August 2006 17:53 Go to previous messageGo to next message
awildeep  is currently offline awildeep
Messages: 105
Registered: May 2006
Location: Buffalo, NY
Senior Member
Tadis,
I do not know a lot about MSSQL, so you may need to find someone who does.

You can get the SQL that was executed on your database from either the dev log file, or the control panel.

Try running it directly on the DB.

My guess is that the DB will not return as fast as you would like. I do not know if MSSQL has an EXPLAIN, or EXPLAIN ANALYZE function but if it does I highly recomend looking at the execution path.

Ultimately 5000+ records should not be a problem for any real database server (Access or FileMaker may have issues depending on archetecture) and from what I understand MSSQL has a descent following.

Here is a list of what I would check (in the order I would check them):
  1. Bad/non-existant indexes. As a test, try indexing everything you are searching by.
  2. Unoptimal SQL path (JOIN ordering, WHERE cluases, GROUP BYs, etc)
  3. Database server may not optimized
  4. Check your database server for hardware issues
Re: Paging with large results set [message #10386 is a reply to message #10304 ] Fri, 18 August 2006 10:43 Go to previous messageGo to next message
halfer  is currently offline halfer
Messages: 9535
Registered: January 2006
Location: West Midlands, UK
Faithful Member
Have you got components on your page? There is a bug somewhere in symfony that repeats the execution of components for every row displayed in your pager, I think. Which would seriously slow your exec times down!

Have a look in your debug toolbar to see where the problems are occuring. As per awildeep's suggestion, try running your query in Enterprise Manager to see if the sql itself is slow.

HTH.


Remember Palestine
Re: Paging with large results set [message #10418 is a reply to message #10386 ] Fri, 18 August 2006 14:56 Go to previous messageGo to next message
Draven  is currently offline Draven
Messages: 760
Registered: January 2006
Faithful Member
halfer wrote on Fri, 18 August 2006 10:43

Have you got components on your page? There is a bug somewhere in symfony that repeats the execution of components for every row displayed in your pager, I think. Which would seriously slow your exec times down!

Have a look in your debug toolbar to see where the problems are occuring. As per awildeep's suggestion, try running your query in Enterprise Manager to see if the sql itself is slow.

HTH.


Actually I found that bug, which is really only setting the compoent variable again in each view instances. If you look in your cache at the config file that holds your view for a component, you'll see that there's a line in there that sets the variable and then prints to the log, so the component isn't really being run multiple times, just the variable and log inoput are being repeated each time that views config file is loaded. Not really a drag on performance, but does appear as if it were.
Re: Paging with large results set [message #10420 is a reply to message #10304 ] Fri, 18 August 2006 15:00 Go to previous messageGo to next message
halfer  is currently offline halfer
Messages: 9535
Registered: January 2006
Location: West Midlands, UK
Faithful Member
Aha, thanks for the info. Have you raised a ticket? If not I am happy to, though you would probably explain it better than I.


Remember Palestine
Re: Paging with large results set [message #10422 is a reply to message #10420 ] Fri, 18 August 2006 15:15 Go to previous messageGo to next message
Draven  is currently offline Draven
Messages: 760
Registered: January 2006
Faithful Member
halfer wrote on Fri, 18 August 2006 15:00

Aha, thanks for the info. Have you raised a ticket? If not I am happy to, though you would probably explain it better than I.


I didn't bother since it's really just the weblogger that keeps repeating and in production mode that is turned off. But you can open one if you like.

I actually ended up writing my own partial_helper and sfView because I didn't like tha way component_slots could only handle one component and thought the term SLOTS should mean it could handle multiple components. So I rewrote it and during that is when I found this "bug".
Re: Paging with large results set [message #10425 is a reply to message #10304 ] Fri, 18 August 2006 15:22 Go to previous messageGo to next message
halfer  is currently offline halfer
Messages: 9535
Registered: January 2006
Location: West Midlands, UK
Faithful Member
OK, I'll raise a ticket... best to iron out even the little buglets. Meanwhile, might your custom sfView be of use to anyone else, do you reckon? Very Happy


Remember Palestine
Re: Paging with large results set [message #10427 is a reply to message #10425 ] Fri, 18 August 2006 15:46 Go to previous messageGo to next message
Draven  is currently offline Draven
Messages: 760
Registered: January 2006
Faithful Member
halfer wrote on Fri, 18 August 2006 15:22

OK, I'll raise a ticket... best to iron out even the little buglets. Meanwhile, might your custom sfView be of use to anyone else, do you reckon? Very Happy



It's a bit "sloppy" at the moment, but I'll try to clean it up and release it as a plugin. I find myself creating more and more plugins these days Smile
Re: Paging with large results set [message #10431 is a reply to message #10304 ] Fri, 18 August 2006 16:32 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
thank you all for your options. I've looked at the dev toolbar and I've realised that the performance problems aren't being caused by the pager or the number of records but by the way in which the query is executed, as such I've tried to play with the propel in the action to try and get the query as I'd like, but to no avail.

Because of the complex nature of the query (3 sub-queries and multiple MAX DISTINCTs) I don't think it will be possible to use propel and as such have decided to use a raw sql as follows:

$con = Propel::getConnection(); 
$sql = 'Select blah blah';

$stmt = $con->prepareStatement($sql); 
$rs = $stmt->executeQuery(ResultSet::FETCHMODE_ASSOC);
$this->rs = $rs;


and in the template, I have:
foreach($rs as $row) {
 foreach($row as $item) {
  echo $item;
 }
}


The resultset does not map to any existing models/peers but I'd like to know how I would go about implementing hydrating/paging the results.

Any help would be greatly appreciated.
Re: Paging with large results set [message #10934 is a reply to message #10304 ] Fri, 25 August 2006 17:38 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
Hi all,

Not sure if anyone is interested but I thought I'd post the workaround that I've come up with.

The problem that I faced was trying to get propel to build a slightly advanced SQL query. I found that it was easier to create a raw sql query but then I lost the wonderful benefits of symfony/propel allows object orientated paging, searching etc..

What I've done as a result is to create a 'View' in MS SQL Server (not sure how this is done on MySQL - if you're familiar with MS Access at all it is like creating a saved query) and updated the database schema xml file in project\config\ to include a new table that mapped to the 'View' created on the SQL Server.

then I proceeded to build-model, generate crud and now I have been able to implement an advanced query that (at least with my experience) seems too complex to produce through propel but with all of the advantages of propel/symfony

please feel free to contact me if you'd like any more info or if you'd like me to clarify exactly what I did.

Tadis
Re: Paging with large results set [message #12320 is a reply to message #10934 ] Mon, 18 September 2006 14:08 Go to previous messageGo to next message
sojiro  is currently offline sojiro
Messages: 1
Registered: September 2006
Junior Member
Hello,
i wanted to use the view in order to make sql query.

I have created the view, but i don't understand how to put it in schema.xml.

Can anyone help me please ?
Re: Paging with large results set [message #12455 is a reply to message #12320 ] Wed, 20 September 2006 10:28 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
Hi there,

to create the view, simply add the view details as if it were another table in the schema.xml file, for example, if your SQL Server View has been named:

'vwContacts'

with the columns:

ContactID, ContactName, ContactTel

then, in your schema.xml file you can define another table as follows:
<table name="vwContacts" idMethod="native" phpName="vwContact">
 <column name="ContactID" type="integer" required="true" primaryKey="true" autoIncrement="true" />
 <column name="ContactName" type="varchar" size="255" />
 <column name="ContactTel" type="varchar" size="50" />
</table>


follwing this, rebuild your models and create a crud for vwContact and you should have what you are looking for. A light issue to be aware of is that, after generating a crud, you may be unable to create/insert depending upon your database structure.

hope this helps
Re: Paging with large results set [message #16419 is a reply to message #10304 ] Thu, 23 November 2006 14:24 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
Going back to my original problem, it seems that the excess time is caused by the pager:

I have a MSQQL View (similar to table and in any case accessible in the same way as a table - see above explanation) defined in the symfony schema/model with 5000+ rows and when I access it through:

$c = new Criteria();
$this->vwList = vwListPeer::doSelect($c);


the page loads in 9-15 seconds, but, if I try doing the same thing through pager:

$c = new Criteria();
$pager = new sfPropelPager('vwList', 10);
$pager->setCriteria($c);
$pager->setPage($this->getRequestParameter('page', 1));
$pager->init();
return $pager;


the page loads in 19-40 seconds.

looking at the debug bar when the pager is used:

+9886 	Creole	prepareStatement(): SELECT COUNT(vwList.ID) FROM vwList

which seems reasonsable, but:
+9499 	PHPView	execute view for template "listSuccess.php"
+8863 	PHPView	decorate content with "sf_app_dir\templates/layout.php"


i'm not using anything (ajax/components/partials) other than the standard symfony module/action

any ideas how this can be improved?
Re: Paging with large results set [message #16436 is a reply to message #10304 ] Thu, 23 November 2006 16:04 Go to previous messageGo to next message
Draven  is currently offline Draven
Messages: 760
Registered: January 2006
Faithful Member
Not sure what exactly the problem is. I have a table with over 800,000 records in it. I use a pager and even in dev mode it takes less than 2 secs to load. Maybe you need to optimize the table?

index.php/fa/195/0/

  • Attachment: query.png
    (Size: 22.95KB, Downloaded 886 time(s))

Re: Paging with large results set [message #16449 is a reply to message #16436 ] Thu, 23 November 2006 18:50 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
Hi Draven

I was going to say that it might have something to do with the version of symfony (I noticed you are using a later version than my 0.6.3 installation), but, I've just ported all the data away from MSSQL into a local MySQL database and the speed increase is staggering!

I don't mean to say that SQL Server is slow - it just looks like MySQL is better optimised for php - hence the data from MSSQL loading fine in MS Access of on asp.net.

only problem is that we use SQL server at work and I'm not sure if the powers that be will like the idea of porting the database server to mysql Confused

thanks for your help anyway
Re: Paging with large results set [message #16450 is a reply to message #10304 ] Thu, 23 November 2006 19:04 Go to previous messageGo to next message
halfer  is currently offline halfer
Messages: 9535
Registered: January 2006
Location: West Midlands, UK
Faithful Member
I prefer to use MySQL for license reasons, but there's nothing wrong with MSSQL in this regard. Copy the queries to your enterprise manager, and see if they run poorly there too. If they are OK there, then write a quick standalone app using the native PHP calls, as the bridge between PHP and MSSQL could be the problem.


Remember Palestine
Re: Paging with large results set [message #16474 is a reply to message #16450 ] Fri, 24 November 2006 11:57 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
tried the same (and other more complex) queries in SQL Studio and load time is minimal 2-5 seconds at most.

I will try doing a native MSSQL connection from php and report the results here.
Re: Paging with large results set [message #16477 is a reply to message #12455 ] Fri, 24 November 2006 12:00 Go to previous messageGo to next message
donharold  is currently offline donharold
Messages: 99
Registered: May 2006
Member
Tadis wrote on Wed, 20 September 2006 10:28

Hi there,

to create the view, simply add the view details as if it were another table in the schema.xml file, for example, if your SQL Server View has been named:

'vwContacts'

with the columns:

ContactID, ContactName, ContactTel

then, in your schema.xml file you can define another table as follows:
<table name="vwContacts" idMethod="native" phpName="vwContact">
 <column name="ContactID" type="integer" required="true" primaryKey="true" autoIncrement="true" />
 <column name="ContactName" type="varchar" size="255" />
 <column name="ContactTel" type="varchar" size="50" />
</table>


follwing this, rebuild your models and create a crud for vwContact and you should have what you are looking for. A light issue to be aware of is that, after generating a crud, you may be unable to create/insert depending upon your database structure.

hope this helps



While that is the correct way, you should go a few additional steps:

1) Your table node should have an attribute "skipSql=true". That way, no SQL will be created so Symfony doesn't attempt to create the table when executing insert-sql.

2) Your table node should have an attribute "readOnly=true". Propel will then prevent the generated object from trying to write to your view, if somebody ever uses it wrong.

3) Add the SQL file to create your view to sqldb.map in your data/sql directory, so the view is created automagically when running insert-sql.

After doing this, you will be able to transparently work with your model and the symfony command line, without the need to bother with manual DB manipulation etc.

[Updated on: Fri, 24 November 2006 12:08]

Re: Paging with large results set [message #16483 is a reply to message #16450 ] Fri, 24 November 2006 13:00 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
just tried the following:

<?
	$connection = mssql_connect( "127.0.0.1",[user],"[pass]")
	or die ( "Sorry - unable to connect to MS SQL" );
	echo( "Congratulations - you are connected to MS SQL" );
	$db = mssql_select_db('[db]',$connection)
	or die ("Sorry database not found");
	$result = mssql_query("select * from vwList");
?>
	<table>
<?
	for ($i = 0; $i < mssql_num_rows($result); ++$i) {
		$row = mssql_fetch_row($result);
?>
	<tr>
<?
	foreach ($row as $field)
	{
		echo '<td>'.$field.'</td>';
	}
?>
	</tr>
<?
	}
?>
	</tbody>
	</table>


and still taking in excess of 30 seconds to pull 5000ish rows - also the SQL Server process is showing 100% CPU usage during this time.

as stated before, running the select * from' through sql studio takes 2-3 seconds most.

Interestingly, tried connecting to the mssql db through odbc:
<?
	$odbc['dsn'] = '[dsn]';
	$odbc['user'] = '[user]';
	$odbc['password'] = '[pass]';

	$conn = odbc_connect($odbc['dsn'],$odbc['user'],$odbc['password'])
		or exit("<h2>Cannot connect to DSN " . $odbc['dsn'] . "</h2>");
	if($conn) { echo "<h2>ODBC Connection Successful</h2>"; }
	$odbcquery = "SELECT * from vwList";
	$odbcexec = odbc_do($conn, $odbcquery);
	odbc_result_all($odbcexec, "");
	odbc_close($conn); 
?>

this is even slower than through the mssql connection and it also results in 100% cpu usage for sqlserver process??

tried connecting through the same odbc dsn in MS Access (linked-tables) and a lightning fast connection??

looks like sql server issue but have no ideas about how to go about fixing it - any help would be greatly appreciated
Re: Paging with large results set [message #16484 is a reply to message #10304 ] Fri, 24 November 2006 13:07 Go to previous messageGo to next message
halfer  is currently offline halfer
Messages: 9535
Registered: January 2006
Location: West Midlands, UK
Faithful Member
http://www.google.co.uk/search?hl=en&q=php+mssql+slow


Remember Palestine
Re: Paging with large results set [message #16494 is a reply to message #16484 ] Fri, 24 November 2006 15:14 Go to previous messageGo to next message
Tadis  is currently offline Tadis
Messages: 15
Registered: August 2006
Location: UK
Junior Member
yes tried the exact same search and have been through the results with no joy I'm afraid - just thought that someone here might have come across the same issue
Re: Paging with large results set [message #17164 is a reply to message #10304 ] Sun, 03 December 2006 13:01 Go to previous messageGo to next message
kubens  is currently offline kubens
Messages: 89
Registered: February 2006
Location: Germany
Member
I got the same unacceptable performance for a table which contains currently approx 4000 rows. But the bottleneck was not the fact that the table contains 4000 rows or more, the bottleneck was the fact that I joined 5 additional tables and Creole needs definitely a lot of time therefore. The aspect with the view was a good hint, but unfortunatelly I have not MySQL 5. I decided to use instead of Creoles ORM native sql statements. I was able to reduce the execution time from ~2000ms to ~400ms. Of course I lost all the advanteges of Creoles ORM, but in this case it was absolutely necessary to realize best performance.

Br
Wolfgang
Re: Paging with large results set [message #17861 is a reply to message #10304 ] Thu, 14 December 2006 00:39 Go to previous message
JoSsiF  is currently offline JoSsiF
Messages: 51
Registered: June 2006
Location: Germany
Member
@kubens: Maybe you could have achieved nearly the same result when using a view, just like Tadis did. I wrote a pretty large application the last months, and I'm using views extensively, since the complexity of some of my queries could hardly be handled by propel. Furthermore, subqueries in propel are impossibly slow, so you need to use something like joins, native SQL or just a view in any case you have to do a listing.

JoSsiF Cool
Previous Topic:how can I change name of "action.class.php"
Next Topic:Problem with _sf_ and requests
Goto Forum:
  

powered by FUDforum - copyright ©2001-2004 FUD Forum Bulletin Board Software