| Paging with large results set [message #10304] |
Thu, 17 August 2006 12:01  |
Tadis Messages: 15 Registered: August 2006 Location: UK |
Junior Member |
|
|
Hi All,
I'm currently using pagination successfully on an archive system, however due to the number of records being input (5000+) I've noticed that performance has degraded somewha. As such, I was wondering if it would be possible to improve the performance of the pager?
The backend database is MSSQL 2000 and I've limited the pager to show only 8 recorrds, but it seems to take an awfully long time to load.
Any suggestions would be greatly appreciated.
|
|
|
| Re: Paging with large results set [message #10347 is a reply to message #10304 ] |
Thu, 17 August 2006 17:53   |
awildeep Messages: 105 Registered: May 2006 Location: Buffalo, NY |
Senior Member |
|
|
Tadis,
I do not know a lot about MSSQL, so you may need to find someone who does.
You can get the SQL that was executed on your database from either the dev log file, or the control panel.
Try running it directly on the DB.
My guess is that the DB will not return as fast as you would like. I do not know if MSSQL has an EXPLAIN, or EXPLAIN ANALYZE function but if it does I highly recomend looking at the execution path.
Ultimately 5000+ records should not be a problem for any real database server (Access or FileMaker may have issues depending on archetecture) and from what I understand MSSQL has a descent following.
Here is a list of what I would check (in the order I would check them):
- Bad/non-existant indexes. As a test, try indexing everything you are searching by.
- Unoptimal SQL path (JOIN ordering, WHERE cluases, GROUP BYs, etc)
- Database server may not optimized
- Check your database server for hardware issues
|
|
|
| Re: Paging with large results set [message #10386 is a reply to message #10304 ] |
Fri, 18 August 2006 10:43   |
halfer Messages: 9535 Registered: January 2006 Location: West Midlands, UK |
Faithful Member |
|
|
Have you got components on your page? There is a bug somewhere in symfony that repeats the execution of components for every row displayed in your pager, I think. Which would seriously slow your exec times down!
Have a look in your debug toolbar to see where the problems are occuring. As per awildeep's suggestion, try running your query in Enterprise Manager to see if the sql itself is slow.
HTH.
Remember Palestine
|
|
|
|
| Re: Paging with large results set [message #10420 is a reply to message #10304 ] |
Fri, 18 August 2006 15:00   |
halfer Messages: 9535 Registered: January 2006 Location: West Midlands, UK |
Faithful Member |
|
|
Aha, thanks for the info. Have you raised a ticket? If not I am happy to, though you would probably explain it better than I.
Remember Palestine
|
|
|
|
|
|
|
|
|
| Re: Paging with large results set [message #12455 is a reply to message #12320 ] |
Wed, 20 September 2006 10:28   |
Tadis Messages: 15 Registered: August 2006 Location: UK |
Junior Member |
|
|
Hi there,
to create the view, simply add the view details as if it were another table in the schema.xml file, for example, if your SQL Server View has been named:
'vwContacts'
with the columns:
ContactID, ContactName, ContactTel
then, in your schema.xml file you can define another table as follows:
<table name="vwContacts" idMethod="native" phpName="vwContact">
<column name="ContactID" type="integer" required="true" primaryKey="true" autoIncrement="true" />
<column name="ContactName" type="varchar" size="255" />
<column name="ContactTel" type="varchar" size="50" />
</table>
follwing this, rebuild your models and create a crud for vwContact and you should have what you are looking for. A light issue to be aware of is that, after generating a crud, you may be unable to create/insert depending upon your database structure.
hope this helps
|
|
|
|
|
|
| Re: Paging with large results set [message #16450 is a reply to message #10304 ] |
Thu, 23 November 2006 19:04   |
halfer Messages: 9535 Registered: January 2006 Location: West Midlands, UK |
Faithful Member |
|
|
I prefer to use MySQL for license reasons, but there's nothing wrong with MSSQL in this regard. Copy the queries to your enterprise manager, and see if they run poorly there too. If they are OK there, then write a quick standalone app using the native PHP calls, as the bridge between PHP and MSSQL could be the problem.
Remember Palestine
|
|
|
|
| Re: Paging with large results set [message #16477 is a reply to message #12455 ] |
Fri, 24 November 2006 12:00   |
donharold Messages: 99 Registered: May 2006 |
Member |
|
|
| Tadis wrote on Wed, 20 September 2006 10:28 | Hi there,
to create the view, simply add the view details as if it were another table in the schema.xml file, for example, if your SQL Server View has been named:
'vwContacts'
with the columns:
ContactID, ContactName, ContactTel
then, in your schema.xml file you can define another table as follows:
<table name="vwContacts" idMethod="native" phpName="vwContact">
<column name="ContactID" type="integer" required="true" primaryKey="true" autoIncrement="true" />
<column name="ContactName" type="varchar" size="255" />
<column name="ContactTel" type="varchar" size="50" />
</table>
follwing this, rebuild your models and create a crud for vwContact and you should have what you are looking for. A light issue to be aware of is that, after generating a crud, you may be unable to create/insert depending upon your database structure.
hope this helps
|
While that is the correct way, you should go a few additional steps:
1) Your table node should have an attribute "skipSql=true". That way, no SQL will be created so Symfony doesn't attempt to create the table when executing insert-sql.
2) Your table node should have an attribute "readOnly=true". Propel will then prevent the generated object from trying to write to your view, if somebody ever uses it wrong.
3) Add the SQL file to create your view to sqldb.map in your data/sql directory, so the view is created automagically when running insert-sql.
After doing this, you will be able to transparently work with your model and the symfony command line, without the need to bother with manual DB manipulation etc.
[Updated on: Fri, 24 November 2006 12:08]
|
|
|
| Re: Paging with large results set [message #16483 is a reply to message #16450 ] |
Fri, 24 November 2006 13:00   |
Tadis Messages: 15 Registered: August 2006 Location: UK |
Junior Member |
|
|
just tried the following:
<?
$connection = mssql_connect( "127.0.0.1",[user],"[pass]")
or die ( "Sorry - unable to connect to MS SQL" );
echo( "Congratulations - you are connected to MS SQL" );
$db = mssql_select_db('[db]',$connection)
or die ("Sorry database not found");
$result = mssql_query("select * from vwList");
?>
<table>
<?
for ($i = 0; $i < mssql_num_rows($result); ++$i) {
$row = mssql_fetch_row($result);
?>
<tr>
<?
foreach ($row as $field)
{
echo '<td>'.$field.'</td>';
}
?>
</tr>
<?
}
?>
</tbody>
</table>
and still taking in excess of 30 seconds to pull 5000ish rows - also the SQL Server process is showing 100% CPU usage during this time.
as stated before, running the select * from' through sql studio takes 2-3 seconds most.
Interestingly, tried connecting to the mssql db through odbc:
<?
$odbc['dsn'] = '[dsn]';
$odbc['user'] = '[user]';
$odbc['password'] = '[pass]';
$conn = odbc_connect($odbc['dsn'],$odbc['user'],$odbc['password'])
or exit("<h2>Cannot connect to DSN " . $odbc['dsn'] . "</h2>");
if($conn) { echo "<h2>ODBC Connection Successful</h2>"; }
$odbcquery = "SELECT * from vwList";
$odbcexec = odbc_do($conn, $odbcquery);
odbc_result_all($odbcexec, "");
odbc_close($conn);
?>
this is even slower than through the mssql connection and it also results in 100% cpu usage for sqlserver process??
tried connecting through the same odbc dsn in MS Access (linked-tables) and a lightning fast connection??
looks like sql server issue but have no ideas about how to go about fixing it - any help would be greatly appreciated
|
|
|
|
|
|
|