| Author | Post |
|---|
are_aaron
Member
| Joined: | Mon Nov 17th, 2008 |
| Location: | |
| Posts: | 2 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | |
| Status: |
Offline
|
| Mana: |  |
|
Posted: Mon Nov 17th, 2008 06:27 pm |
|
| 2 of my friends and I took the building design and construction systems test a when 4.0 first started. We all felt great about the vignettes ..... we were all marked off on the vignettes and did not pass. We received the initial results like 2 months ago, just last friday we all received "reprints" which graded us all different (all still fail, but all different). My faith in the system was already a little shaky. Has this happened to anyone else? Have you gotten any reasonable explanation?
|
xcentric
Member
| Joined: | Fri Mar 28th, 2008 |
| Location: | |
| Posts: | 429 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | LEED and 6 passed, waiting on BD results. |
| Status: |
Offline
|
| Mana: |   |
|
Posted: Mon Nov 17th, 2008 08:18 pm |
|
| What do you mean by "graded different"? The only feedback I've heard is Pass, Fail, and "major deficiencies" on a specific vignette. Did the reprint change which vignettes had deficiencies or did you get other details?
|
are_aaron
Member
| Joined: | Mon Nov 17th, 2008 |
| Location: | |
| Posts: | 2 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | |
| Status: |
Offline
|
| Mana: | |
|
Posted: Mon Nov 17th, 2008 08:37 pm |
|
| yeah, i was marked off "moderate deficiencies" on all three vignettes at first. Then on the reprint I was marked "major deficiencies" on just the ramp, and no marks on anything else. The 'fail' didn't change.
|
oakdale
Member
| Joined: | Fri Jun 13th, 2008 |
| Location: | New York, New York USA |
| Posts: | 81 |
| Exams Taken: | PD, CD, SP, BP | | Exams Passed: | PD, CD, SP | | Describes Me: | mailbox watcher |
| Status: |
Online
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 03:11 am |
|
I got a reprint today for Schematic Design. I still failed, but on the original I had deficiencies on both vignettes, and on the letter i received today, i only had deficiencies on the interior layout.
I know i made mistakes on both parts, on the interior layout, i moved a wall, and the door disappeared, right as time was running out. On the building layout vignette, i had one room that was supposed to have an exterior window, but i discovered my mistake too late and didn't have enough time to move enough elements around to get it there.
Basically what the reprint told me today is that the test is now graded easier than it was initially.
|
brudgers
Member
| Joined: | Thu Jun 1st, 2006 |
| Location: | The Moment |
| Posts: | 7639 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | Newly described. |
| Status: |
Offline
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 02:48 pm |
|
I see two likely scenarios:
- There was a computer glitch which caused one candidate's score to be reported as another candidate's...ie. there was a misprinted letter.
- More likely, a group of new test vignettes were statisicaly determined to be more difficult than the test specification, and therefore the scoring algorithm for those vignettes was adjusted to require only the same competency as a typical vignette.
The difficulty for NCARB in developing the test vignettes is that there is no way to pre-flight the vignettes without scoring it. This prevents NCARB from establishing the validity of the vignette as a measurement instrument prior to putting it into the test rotation.
For the original poster, it is not uncommon for people to fail the graphic vignettes particularly when they have not posted solutions in this forum. The vignettes can readily be solved, but they do have their quirks with which real world experience often conflicts.
|
Spellcaster
Architect
| Joined: | Thu Apr 15th, 2004 |
| Location: | ARCHITECT, LEED AP, USA |
| Posts: | 959 |
| Exams Taken: | PD, GS, LF, ME, BD/MM, CD, SP, BP, BT | | Exams Passed: | PD, GS, LF, ME, BD/MM, CD, SP, BP, BT | | Describes Me: | |
| Status: |
Offline
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 03:32 pm |
|
The original poster's test was graded more harshly the second time.
The other poster, who took a different test, was graded more leniently the second time.
That sounds like a bigger problem than just one group of test vignettes needing to be recalibrated - and a computer glitch swapping results is also unlikely as this is affecting more than one test division. (Not impossible, but previous computer glitches have always affected just one test division - like the time that a bunch of SP tests were mis-reported as fails, or the time that a batch of ME results were transposed.)
It seems like NCARB may have rushed to get result letters out to candidates as quickly as possible - to avoid a repeat of the 2004 uproar over the 4-month delay - while they were perhaps still performing across-the-board graphic cut score/calibration studies.
|
brudgers
Member
| Joined: | Thu Jun 1st, 2006 |
| Location: | The Moment |
| Posts: | 7639 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | Newly described. |
| Status: |
Offline
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 03:47 pm |
|
Actually a computer glitch with the 4.0 scoring may be a greater possibility than I thought.
The combination of graphics and multiple choice into a single test probably required a major rewrite of the scoring software.
A bug in coordinating between the two would make itself apparent across multiple divisions.
It seems like a bug in the "report writing" portion of NCARB's process software.
|
Spellcaster
Architect
| Joined: | Thu Apr 15th, 2004 |
| Location: | ARCHITECT, LEED AP, USA |
| Posts: | 959 |
| Exams Taken: | PD, GS, LF, ME, BD/MM, CD, SP, BP, BT | | Exams Passed: | PD, GS, LF, ME, BD/MM, CD, SP, BP, BT | | Describes Me: | |
| Status: |
Offline
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 03:54 pm |
|
Could be.
It would be a weird bug to mis-report only some vignettes on some tests, and to mis-report them in both directions (better and worse than actual performance.)
I'm more inclined to think that the scoring committee made changes to the thresholds after some reports had already been sent.
But we'll probably never get the real story from NCARB.
|
oakdale
Member
| Joined: | Fri Jun 13th, 2008 |
| Location: | New York, New York USA |
| Posts: | 81 |
| Exams Taken: | PD, CD, SP, BP | | Exams Passed: | PD, CD, SP | | Describes Me: | mailbox watcher |
| Status: |
Online
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 04:33 pm |
|
The letter itself says:
It has come to our attention that the score report had some missing information in terms of either the diagnostic inofrmation about your weak areas or the rolling clock summary sheet.
I've attached a pdf so you can see for yourselves. Sorry it's blurry, i don't know what's going on with my scanner these days.
Attachment: Reprint.pdf (Downloaded 62 times)
|
kmonkey10
Member
| Joined: | Mon Aug 7th, 2006 |
| Location: | Nevada USA |
| Posts: | 119 |
| Exams Taken: | PD, GS, LF, CD, SP | | Exams Passed: | PD, SP | | Describes Me: | tired...just plain tired... |
| Status: |
Offline
|
| Mana: | |
|
Posted: Tue Nov 18th, 2008 07:32 pm |
|
| I received a similar letter for the Structures exam I took. However, the content areas that I failed did not change, nor did the asterisk next to one of them. The original result letter and the new result letter are exactly the same (with the exception of one of my later exams being in the reporting box in the lower left corner.)
|
Coach
Member
| Joined: | Tue Mar 4th, 2008 |
| Location: | USA |
| Posts: | 5812 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | Architect |
| Status: |
Offline
|
| Mana: | |
|
Posted: Wed Nov 19th, 2008 09:44 pm |
|
| As the letter states, the original reports just omitted details. This is not surprising for a new system.
|
FinitoCompleto
Member
| Joined: | Wed Jun 7th, 2006 |
| Location: | |
| Posts: | 258 |
| Exams Taken: | | | Exams Passed: | | | Describes Me: | |
| Status: |
Offline
|
| Mana: | |
|
Posted: Wed Nov 19th, 2008 11:34 pm |
|
But some of these posts are indicating changed details, not just omitted details:
yeah, i was marked off "moderate deficiencies" on all three vignettes at first. Then on the reprint I was marked "major deficiencies" on just the ramp, and no marks on anything else...
|
Current time is 07:53 pm | |
|
