Showing posts with label scheduled. Show all posts
Showing posts with label scheduled. Show all posts

Monday, March 26, 2012

Killer problem with Scheduled Reports

I'm running on SQL 2000 with SRS. Standard install (I think, it was all set up by IT)

I have reports that run fine in the Report Manager but when they are run through a schedule (email) some of them (albeit most) "rsProcessingAborted" because of a report error.

Looking at the logs it seems generally based around comparison failure because of datatypes, eg (from the logs):

Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: The processing of sort expression for the list 'List1' cannot be performed. The comparison failed. Please check the data type returned by sort expression. Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: The processing of group expression for the table 'tblGroupTable' cannot be performed. The comparison failed. Please check the data type returned by group expression.

Now, when I remove sorting/grouping the reports run through the schedule OK; but that is obviously NOT what I want the report to look like. But it proves that the sorting/grouping has something to do with the problem.

I'm pretty sure that they are NOT the problem though, as the reports render in Report Manager fine with all the original sorting/grouping in place (ie no problems during development, just when we wanted them served to the employees. Isn't that always the way it goes?). We've even had some reports which scheduled OK previously, but throw errors now.

I'm at a loss!!

Could it be a time-out issue somewhere? If so: where? Looking at the Execution Logs, the reports that fail have unusually long Process Times, with a Data Retrieval Time of 0. I guess that the process time is longer because of processing the error, but does the Data Retrieval time of 0 mean that no data was delivered to the report? How can that be? IT tell me it can't be memory issues as it's sitting on a 2GB machine, but could there be something else with SQL set-up? I googled the error and found a few pages where people said that puting SRS in its own application pool fixed their problem. When I passed this on to IT they said that would only help memory... Any ideas?

Any suggestions greatfully appreciated

Do the report subscriptions ALWAYS fail, or intermittently?

Do those same reports ALWAYS succeed when executed live?

How much RAM is on the Dev machine, and what OS?

What OS is on the productio machine?

What changed between the time that some scheduled reports worked fine, and when they started erroring?

Removing the grouping/sorting may have alleviated the load on the processing engine enough so that the report succeeded.

|||Also, what are the types and #'s of CPU's on the Dev and Production machines?|||

Mike Schetterer -- MSFT wrote:

Do the report subscriptions ALWAYS fail, or intermittently?

The reports seem to fail always when they contain some data. Maybe the oposite is better: The reports don't fail when they do not contain data. Of the report I can remember off the top of my head: the schedule didn't fail on 4 occasions, three of those didn't have any data, one had one record.

For example, for one of our reports that is meant to run this morning:

Time Start

Time End

Data Retrieval

Process

Render

Status

User Name

pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=3

6/02/2006 7:00:08 AM

6/02/2006 7:00:09 AM

0

112

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=8

6/02/2006 7:00:08 AM

6/02/2006 7:00:12 AM

2979

359

255

rsSuccess

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=6

6/02/2006 7:00:57 AM

6/02/2006 7:00:58 AM

0

47

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=11

6/02/2006 7:00:57 AM

6/02/2006 7:00:58 AM

0

60

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=13

6/02/2006 7:00:58 AM

6/02/2006 7:00:58 AM

0

51

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=1

6/02/2006 7:00:58 AM

6/02/2006 7:00:58 AM

0

72

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=9

6/02/2006 7:00:59 AM

6/02/2006 7:00:59 AM

0

63

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE


pmtStartDate=01/01/2000 00:00:00
pmtEndDate=02/06/2006 00:00:00
Team=10

6/02/2006 7:00:59 AM

6/02/2006 7:00:59 AM

0

48

0

rsProcessingAborted

NT AUTHORITY\NETWORK SERVICE

Mike Schetterer -- MSFT wrote:

Do those same reports ALWAYS succeed when executed live?

YES!

Mike Schetterer -- MSFT wrote:

How much RAM is on the Dev machine, and what OS?

Intel Xeon 2.4Gb x 2, 2Gb Ram

The Dev machine and the production machine are the same. I use a Remote Desktop into the production machine from my desktop, I guess the documents are stored on my workstation and deployed to the production machine.

Mike Schetterer -- MSFT wrote:

What OS is on the productio machine?

Windows Server 2003, SP1

Mike Schetterer -- MSFT wrote:

What changed between the time that some scheduled reports worked fine, and when they started erroring?

Nothing that I am aware of... but I'll ask the IT people...

Mike Schetterer -- MSFT wrote:

Removing the grouping/sorting may have alleviated the load on the processing engine enough so that the report succeeded.

The schedules fail regardless of the time of day they run (I wondered if the server got overloaded whilst trying to do other things): but there is no difference.

Another interesting fact: whilst investigating I discovered that one of the reports that ran through a schedule after all grouping and sorting was removed displayed "#error" in a cell that contained the result from a Library I'd written. Though this report runs and displays fine through the Report Manager! (this field was not used in the grouping or sorting either).

Yes, I've read somewhere else that how you've configured the Application Pool can make a difference (our IT said it wouldn't here). But I've definitely read in a different forum that that fixed their problem.

At the moment the Report Server Interface and the Report Server are in the DefaultAppPool.

Thanks for your interest!

Perry

|||

Do your reports have parameters?

When you create your subscription - are the parameters set staticly or are the based on a query. If they're query based - can you try running the query and selecting some of the value combinations you receive in return through the UI to see if they work?

Your comment above about report executions failing when there is data, but succeeding when there is no data - is it true to say that whenever there is data the subscription fails, or just some of the time?

Thanks,

-Lukasz


This posting is provided "AS IS" with no warranties, and confers no rights.

|||

Lukasz Pawlowski -- MS wrote:

Do your reports have parameters?

Some do, others don't, but it doesn't seem to be a problem.

Lukasz Pawlowski -- MS wrote:

When you create your subscription - are the parameters set staticly or are the based on a query. If they're query based - can you try running the query and selecting some of the value combinations you receive in return through the UI to see if they work?

Both, or sometimes all three (included type in)

Yes, ALL the reports run fine through the Report Manager.

Lukasz Pawlowski -- MS wrote:

Your comment above about report executions failing when there is data, but succeeding when there is no data - is it true to say that whenever there is data the subscription fails, or just some of the time?

No: they fail most of the time. In the section of the ExecutionLog in a previous post you can see that for that report it ran once (with data) and failed the rest (they would have had data too.

Thanks,

Perry

|||

What are the group and sort expressions from the report that you find is failing?

-Lukasz

|||

Lukasz Pawlowski -- MS wrote:

What are the group and sort expressions from the report that you find is failing?

In every case it's "=Fields!FieldName.value"

Perry

killed/rollback stuck on object_name(99)

Hello:
I have a process that's been stuck for two days.. It's a stored procedure
that runs as part of a scheduled SqlAgent job. I tried to kill the process
which put it into a rollback. Kill with statusonly returns:
SPID 52: transaction rollback in progress. Estimated rollback completion:
0%. Estimated time remaining: 0 seconds.
I ran dbcc page for the resource that is listed in the wait type
(PAGEIOLATCH_UP)
and it points to Obj_id 99. Running "select object_name(99)" returns the
object name "Allocation".
Does anyone know what this means and how to allow the rollback to complete?
This spid blocks other processess that try to run in the affected database.
Even Enterprise Manager is blocked. Can't refresh table list or procedure
list in EM. Current activity times out. I can use sp_who2 to see active
processes.hi,
Right, try again using this:
KILL <your_process> WITH STATUSONLY
Because of your process has been running a long time the rollback will take
a lot of time (not the same, of course, but a lof anyway)
Rollback is undoing changes and transactions commited
Current location: Alicante (ES)
"tthrone" wrote:

> Hello:
> I have a process that's been stuck for two days.. It's a stored procedure
> that runs as part of a scheduled SqlAgent job. I tried to kill the proces
s
> which put it into a rollback. Kill with statusonly returns:
> SPID 52: transaction rollback in progress. Estimated rollback completion:
> 0%. Estimated time remaining: 0 seconds.
> I ran dbcc page for the resource that is listed in the wait type
> (PAGEIOLATCH_UP)
> and it points to Obj_id 99. Running "select object_name(99)" returns the
> object name "Allocation".
> Does anyone know what this means and how to allow the rollback to complete
?
> This spid blocks other processess that try to run in the affected database
.
> Even Enterprise Manager is blocked. Can't refresh table list or procedure
> list in EM. Current activity times out. I can use sp_who2 to see active
> processes.
>|||Hi Enric,
I did that. It returns:
> SPID 52: transaction rollback in progress. Estimated rollback completion:
> 0%. Estimated time remaining: 0 seconds.
>
It's been returning the same thing for two days. It doesn't appear to be
making any progress on the rollback. The original process should have done
123,000 row inserts on a previously empty table. I can't imagine 123k rows
should take 2 days to rollback. I think it's totally stuck and idle.
"Enric" wrote:
> hi,
> Right, try again using this:
> KILL <your_process> WITH STATUSONLY
> Because of your process has been running a long time the rollback will tak
e
> a lot of time (not the same, of course, but a lof anyway)
> Rollback is undoing changes and transactions commited
> --
> Current location: Alicante (ES)
>
> "tthrone" wrote:
>|||First, try to find out what application and T-SQL statement caused this
situation so perhaps it won't repeat:
DBCC INPUTBUFFER (spid) will display the last T-SQL statement sent by the
client application owning this SPID.
SP_LOCK (spid) will list information about what specific objects the SPID
currently has locked and what type of lock (table, page, etc.).
Next, try to diagnose what is going on with your server hard disks, memory,
etc. that may have caused this unusual cirsumstance. If the server is
running critically low on disk space, this can cause problems when
attempting rollback a large transaction. Also, go into the windows
management console and review the event logs for possible evidence.
This article describes how get more detailed information about the current
status of the SPID:
http://support.microsoft.com/defaul...kb;en-us;171224
For example, the Process Status Structure (PSS) has the following values:
0x4000 -- Delay KILL and ATTENTION signals if inside a critical section
0x2000 -- Process is being killed
0x800 -- Process is in backout, thus cannot be chosen as deadlock victim
0x400 -- Process has received an ATTENTION signal, and has responded by
raising an internal exception
0x100 -- Process in the middle of a single statement transaction
0x80 -- Process is involved in multi-database transaction
0x8 -- Process is currently executing a trigger
0x2 -- Process has received KILL command
0x1 -- Process has received an ATTENTION signal
This article describes how to identify and troubleshoot an orphaned
connection:
http://support.microsoft.com/kb/137983/EN-US/
If the SPID can't be killed, then:
1. stop the SQL Server service (no need to reboot)
2. using Windows Explorer, move the data and transaction log file(s) to
another location
3. re-start the service
4. restore the database from the most recent backup
"tthrone" <tthrone@.discussions.microsoft.com> wrote in message
news:D7361B71-3C5A-41CE-A4D0-68685B910E3E@.microsoft.com...
> Hello:
> I have a process that's been stuck for two days.. It's a stored procedure
> that runs as part of a scheduled SqlAgent job. I tried to kill the
> process
> which put it into a rollback. Kill with statusonly returns:
> SPID 52: transaction rollback in progress. Estimated rollback completion:
> 0%. Estimated time remaining: 0 seconds.
> I ran dbcc page for the resource that is listed in the wait type
> (PAGEIOLATCH_UP)
> and it points to Obj_id 99. Running "select object_name(99)" returns the
> object name "Allocation".
> Does anyone know what this means and how to allow the rollback to
> complete?
> This spid blocks other processess that try to run in the affected
> database.
> Even Enterprise Manager is blocked. Can't refresh table list or procedure
> list in EM. Current activity times out. I can use sp_who2 to see active
> processes.
>|||Thanks JT. I know some answers to questions/issues you listed. I know the
transation that was in-flight, but I don't know why it stuck. Still can't
figure out why it remains stuck, but I found something interesting in the
process of doing some of what you suggested.
For one, I see this spid blocks some of my attempts to use sysobjects. I
mentioned that I can't refresh procedures or tables in EM on this database.
I think that's why. I have it narrowed down to one (maybe a few) affected
tables. I can query sysobjects so long as I don't try to read certain rows.
Not sure how that happened!
I think I'm going to have to try your suggestion about stopping the service
and restoring.
Thanks for the help.
"JT" wrote:

> First, try to find out what application and T-SQL statement caused this
> situation so perhaps it won't repeat:
> DBCC INPUTBUFFER (spid) will display the last T-SQL statement sent by the
> client application owning this SPID.
> SP_LOCK (spid) will list information about what specific objects the SPID
> currently has locked and what type of lock (table, page, etc.).
> Next, try to diagnose what is going on with your server hard disks, memory
,
> etc. that may have caused this unusual cirsumstance. If the server is
> running critically low on disk space, this can cause problems when
> attempting rollback a large transaction. Also, go into the windows
> management console and review the event logs for possible evidence.
> This article describes how get more detailed information about the current
> status of the SPID:
> http://support.microsoft.com/defaul...kb;en-us;171224
> For example, the Process Status Structure (PSS) has the following values:
> 0x4000 -- Delay KILL and ATTENTION signals if inside a critical section
> 0x2000 -- Process is being killed
> 0x800 -- Process is in backout, thus cannot be chosen as deadlock victim
> 0x400 -- Process has received an ATTENTION signal, and has responded by
> raising an internal exception
> 0x100 -- Process in the middle of a single statement transaction
> 0x80 -- Process is involved in multi-database transaction
> 0x8 -- Process is currently executing a trigger
> 0x2 -- Process has received KILL command
> 0x1 -- Process has received an ATTENTION signal
> This article describes how to identify and troubleshoot an orphaned
> connection:
> http://support.microsoft.com/kb/137983/EN-US/
> If the SPID can't be killed, then:
> 1. stop the SQL Server service (no need to reboot)
> 2. using Windows Explorer, move the data and transaction log file(s) to
> another location
> 3. re-start the service
> 4. restore the database from the most recent backup
>
> "tthrone" <tthrone@.discussions.microsoft.com> wrote in message
> news:D7361B71-3C5A-41CE-A4D0-68685B910E3E@.microsoft.com...
>
>|||When querying sysobjects (or any other blocked table), you can get around
the locks by changing the isolation level to read uncommitted data. However,
this should not be used in a production system except perhaps in some
reporting situations.
set transaction isolation level read uncommitted
select * from sysobjects
"tthrone" <tthrone@.discussions.microsoft.com> wrote in message
news:1C4B57D4-1894-4688-8E32-6157E28AD503@.microsoft.com...
> Thanks JT. I know some answers to questions/issues you listed. I know
> the
> transation that was in-flight, but I don't know why it stuck. Still can't
> figure out why it remains stuck, but I found something interesting in the
> process of doing some of what you suggested.
> For one, I see this spid blocks some of my attempts to use sysobjects. I
> mentioned that I can't refresh procedures or tables in EM on this
> database.
> I think that's why. I have it narrowed down to one (maybe a few) affected
> tables. I can query sysobjects so long as I don't try to read certain
> rows.
> Not sure how that happened!
> I think I'm going to have to try your suggestion about stopping the
> service
> and restoring.
> Thanks for the help.
> "JT" wrote:
>|||I normally do set the transaction isolation level to read uncommitted. I di
d
that in this case as well.
I even tried using the hint "with(readuncommitted)" but it was still blocked
by the stuck spid when I tried to return the sysobject rows of tables that
were affected.
Our DBA is going to bounce the service later today. I'm hoping it will
clear up after the restart.
"JT" wrote:

> When querying sysobjects (or any other blocked table), you can get around
> the locks by changing the isolation level to read uncommitted data. Howeve
r,
> this should not be used in a production system except perhaps in some
> reporting situations.
> set transaction isolation level read uncommitted
> select * from sysobjects
> "tthrone" <tthrone@.discussions.microsoft.com> wrote in message
> news:1C4B57D4-1894-4688-8E32-6157E28AD503@.microsoft.com...
>
>

Friday, March 9, 2012

Keeping SQL 2000 databases in synch after restore.

We have 4 different databases, running on sql server 2000. I scheduled
the database to be backed up at 5:00, and every hour for the trans
log.
If we have a hardware failure and have to restore from backup. how do
I make sure that the database are all starting at the same time. (if
database common takes 2 min to back up and database customer takes 30
min to back up) how do I make sure these 2 database are at the same
point in time.
Is there anything I can do in my sql backup job to have the backup's
end at the same time?
Currently all of the backup are different jobs.
Nicholas.GadaczSo you're saying there are transactional integrity reasons why the DB's all
need to be restored to the exact same point in time?
RESTORE LOG allows you to specify STOPAT for point in time recovery, but
RESTORE DATABASE doesn't.
I can't think of a clean way to do that for a full database without using
tran log restores...
--
Brian Moran
Principal Mentor
Solid Quality Learning
SQL Server MVP
http://www.solidqualitylearning.com
"First Tracks Research" <info@.ftresearch.com> wrote in message
news:4d02a84e.0401061017.1db0b24d@.posting.google.com...
> We have 4 different databases, running on sql server 2000. I scheduled
> the database to be backed up at 5:00, and every hour for the trans
> log.
> If we have a hardware failure and have to restore from backup. how do
> I make sure that the database are all starting at the same time. (if
> database common takes 2 min to back up and database customer takes 30
> min to back up) how do I make sure these 2 database are at the same
> point in time.
> Is there anything I can do in my sql backup job to have the backup's
> end at the same time?
> Currently all of the backup are different jobs.
> Nicholas.Gadacz|||I really don't mean to be a smart ass here. I realize it may not be the
answer you are looking for. But this is kind of a design issue. If there
are transactional integrity issues, the easiest and cleanest way is to put
those objects in the same database. You may have name conflicts, but those
will be much easier to resolve now than the issues you have started to run
into now. What if tomorrow you want to send a snapshot replication of five
related tables? You will have the same issue.
If you do take my advice and put your databases into one, a good way to
avoid naming conflicts in your code, is to script one of the databases in
its entirety, do a search and replace and change all occurences of
conflicting names at the object definition as well as the references in code
simultaneously. Then you can put the two together without conflict. Again,
if you decide to do that, I can be made available for assistance remotely.
Pls remove NO_SPAM(s) from my email if you need to reach me.
--
Ata R
Parvan Consulting Inc
NO_SPAMar_alias001@.NO_SPAMparvan.net
"First Tracks Research" <info@.ftresearch.com> wrote in message
news:4d02a84e.0401061017.1db0b24d@.posting.google.com...
> We have 4 different databases, running on sql server 2000. I scheduled
> the database to be backed up at 5:00, and every hour for the trans
> log.
> If we have a hardware failure and have to restore from backup. how do
> I make sure that the database are all starting at the same time. (if
> database common takes 2 min to back up and database customer takes 30
> min to back up) how do I make sure these 2 database are at the same
> point in time.
> Is there anything I can do in my sql backup job to have the backup's
> end at the same time?
> Currently all of the backup are different jobs.
> Nicholas.Gadacz